The outbreak of the pandemic Covid-19 has transformed the education system in most countries worldwide. Following the lockdown measures in Malaysia, the Malaysian education system has fully transformed from conventional learning to online learning or known as e-learning as an alternative to minimize social contacts and physical communication to curb the transmission of Covid-19. In this regard, this study aims to identify the challenges faced by students in higher learning institutions throughout Malaysia during the implementation of the e-learning program. This study is based on a large sampling consisting of 2394 students from both public and private universities. The result from this study is analyzed through inferential methods such as the Spatial Analysis, the Principal Component Analysis, and the Mann-Whitney U test and through descriptive methods using the frequency analysis and the percentage analysis. Findings from this study suggest that location significantly influenced the challenges faced by students throughout the implementation of e-learning in higher learning institutions. For example, students in rural areas which can be identified as "vulnerable groups" are more likely to face both technical and connection with the internet access, tend to have a declining focus on learning and are prone to physical health problems, facing social isolation and low digital literacy compared to students in urban areas. Based on geographical analysis, students in Sabah, Perlis, and Melaka are most at risk of facing e-learning challenges. An anomaly case of students in Kuala Lumpur, however, posed another different result compared to other cities as they confront similar challenges with students in rural areas. This study provides the nuances of location and its implications for vulnerable groups that may put them at disadvantage in the e-learning program. Findings from this study will help to inform the relevant authorities and policymakers in improving the implementation of e-learning in Malaysia, especially towards the vulnerable groups so that it can be delivered more systematically and efficiently.