ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology

• DOI: 10.1016/j.ajog.2023.04.020
• Publication Date: 2023-04-22T23:46:46Z
• Authors: Sarah W. Li; Matthew W. Kemp; Susan J.S. Logan; Pooja Sharma Dimri; Navkaran Singh; Citra N.Z. Mattar; Pradip Dashraath; Harshaana Ramlal; Aniza P. Mahyuddin; Suren Kanayan; Sean W.D. Carter; Serene P.T. Thain; Erin L. Fee; Sebastian E. Illanes; Mahesh A. Choolani; Mary Rauff; Arijit Biswas; Jeffrey J.H. Low; Joseph S. Ng; Arundhati T. Gosavi; Zhongwei Huang; Preethi Rajendran; Kwok Weng Roy Ng; Jeslyn J.L. Wong; Grace M.F. Chan; Kelvin Zhi Xing Lee; Jeannie J.Y. Yap; Nurulhuda Ahmad; Shwetha Shanmugam; Rajeswari Kathirvel; Manisha Mathur; Whui Whui Lim; Min Yu Lim
• Keywords: Interface (matter)

Natural language processing is a form of artificial intelligence that allows human users to interface with machine without using complex codes. The ability natural systems, such as ChatGPT, successfully engage healthcare systems requiring fluid reasoning, specialist data interpretation, and empathetic communication in an unfamiliar evolving environment poorly studied. This study investigated whether the ChatGPT could complete mock objective structured clinical examination simulating assessment for membership Royal College Obstetricians Gynaecologists.This aimed determine additional training, would achieve score at least equivalent achieved by candidates who sat virtual examinations Singapore.This was conducted 2 phases. In first phase, total 7 discussion questions were selected from historical cohorts (cohorts A B) questions. examined these responses recorded script. Of note, (acting anonymizers) on same videoconferencing, their transcribed verbatim into written scripts. 3 sets response scripts mixed, each set allocated 1 actors. second actors used presenting examiners These blind scored 14 qualified examiners. scores unblinded compared candidate performance scores.The average given 77.2%. (n=26 candidates) 73.7 %. demonstrated sizable improvements over several subject domains. median time taken station 2.54 minutes, well before 10 minutes allowed.ChatGPT generated factually accurate contextually relevant answers based settings within very short period. outperformed knowledge areas. Not all able discern between responses. Our highlight emergent models demonstrate reasoning environments compete have undergone extensive training.