ChatGPT’s Performance on the Hand Surgery Self-Assessment Exam: A Critical Analysis

• DOI: 10.1016/j.jhsg.2023.11.014
• Publication Date: 2024-01-02T05:03:17Z
• Authors: Yuri Han; Hassaam S. Choudhry; Michael E. Simon; Brian M. Katt

PurposeTo assess the performance of Chat Generative Pre-Trained Transformer (ChatGPT) when answering self-assessment exam questions in hand surgery and to compare correct results for text-only those that included images.MethodsThis study used 10 exams from 2004 2013 provided by American Society Surgery Hand (ASSH). ChatGPT's on image-based was compared. The primary outcomes were total score, score questions, questions. secondary proportion which ChatGPT additional explanations, length elaborations, number answers with certainty.ResultsOut 1,583 answered 573 (36.2%) correct. performed better than Out 1,127 442 (39.2%) correctly. 456 it 131 (28.7%) There no difference between elaborations among Although there got incorrect, longer 1,441 confident answers, 548 (38.0%) correct; out 142 unconfident 25 (17.6%) correct.ConclusionsChatGPT poorly ASSH 2013. It Even its highest 42% year 2012, AI platform would not have received continuing medical education credit or Board Surgery. only considering without images, high 44% "passed" examination.Clinical relevanceAt this time, professionals, trainees, patients should use caution as program has yet developed proficiency subspecialty knowledge. To images. This certainty. examination.