Brain structural features of healthy individuals are associated with olfactory functions. However, due to the pathophysiological differences, congenital and acquired anosmia may exhibit different structural characteristics. A systematic review was undertaken to compare brain structural features between patients with congenital and acquired anosmia. A systematic search was conducted using PubMed/MEDLINE and Scopus electronic databases to identify eligible reports on anosmia and structural changes and reported according to PRISMA guidelines. Reports were extracted for information on demographics, psychophysical evaluation, and structural changes. Then, the report was systematically reviewed based on various aetiologies of anosmia in relation to (1) olfactory bulb, (2) olfactory sulcus, (3) grey matter (GM), and white matter (WM) changes. Twenty-eight published studies were identified. All studies reported consistent findings with strong associations between olfactory bulb volume and olfactory function across etiologies. However, the association of olfactory function with olfactory sulcus depth was inconsistent. The present study observed morphological variations in GM and WM volume in congenital and acquired anosmia. In acquired anosmia, reduced olfactory function is associated with reduced volumes and thickness involving the gyrus rectus, medial orbitofrontal cortex, anterior cingulate cortex, and cerebellum. These findings contrast to those observed in congenital anosmia, where a reduced olfactory function is associated with a larger volume and higher thickness in parts of the olfactory network, including the piriform cortex, orbitofrontal cortex, and insula. The present review proposes that the structural characteristics in congenital and acquired anosmia are altered differently. The mechanisms behind these changes are likely to be multifactorial and involve the interaction with the environment.