To utilise effectively tools that employ machine learning (ML) in clinical practice medical students and doctors will require a degree of understanding of ML models. To evaluate current levels of understanding, a formative examination and survey was conducted across three centres in Australia, New Zealand and the United States. Of the 245 individuals who participated in the study (response rate = 45.4%), the majority had difficulty with identifying weaknesses in model performance analysis. Further studies examining educational interventions addressing such ML topics are warranted.