The Philippines are central to understanding the expansion of the Austronesian language family from its homeland in Taiwan. It remains unknown to what extent the distribution of Malayo-Polynesian languages has been shaped by back migrations and language leveling events following the initial Out-of-Taiwan expansion. Other aspects of language history, including the effect of language switching from non-Austronesian languages, also remain poorly understood. Here we apply Bayesian phylogenetic methods to a core-vocabulary dataset of Philippine languages. Our analysis strongly supports a sister group relationship between the Sangiric and Minahasan groups of northern Sulawesi on one hand, and the rest of the Philippine languages on the other, which is incompatible with a simple North-to-South dispersal from Taiwan. We find a pervasive geographical signal in our results, suggesting a dominant role for cultural diffusion in the evolution of Philippine languages. However, we do find some support for a later migration of Gorontalo-Mongondow languages to northern Sulawesi from the Philippines. Subsequent diffusion processes between languages in Sulawesi appear to have led to conflicting data and a highly unstable phylogenetic position for Gorontalo-Mongondow. In the Philippines, language switching to Austronesian in 'Negrito' groups appears to have occurred at different time-points throughout the Philippines, and based on our analysis, there is no discernible effect of language switching on the basic vocabulary.