Examining the Performance of ChatGPT 3.5 and Microsoft Copilot in Otolaryngology: A Comparative Study with Otolaryngologists’ Evaluation
Medicine; Surgery; Otolaryngology; Head neck; ChatGPT; Chatbot; Artificial; GPT; Instrument; Internet; Tool; Intelligence; Performance; Comparison; Diagnosis; Treatment
Artificial; Chatbot; ChatGPT; Comparison; Diagnosis; GPT; Head neck; Instrument; Intelligence; Internet; Medicine; Otolaryngology; Performance; Surgery; Tool; Treatment
DOI:
10.1007/s12070-024-04729-1
Publication Date:
2024-05-01T14:48:10Z
AUTHORS (6)
ABSTRACT
To evaluate the response capabilities, in a public healthcare system otolaryngology job competition examination, of ChatGPT 3.5 and an internet-connected GPT-4 engine (Microsoft Copilot) with the real scores of otolaryngology specialists as the control group. In September 2023, 135 questions divided into theoretical and practical parts were input into ChatGPT 3.5 and an internet-connected GPT-4. The accuracy of AI responses was compared with the official results from otolaryngologists who took the exam, and statistical analysis was conducted using Stata 14.2. Copilot (GPT-4) outperformed ChatGPT 3.5. Copilot achieved a score of 88.5 points, while ChatGPT scored 60 points. Both AIs had discrepancies in their incorrect answers. Despite ChatGPT's proficiency, Copilot displayed superior performance, ranking as the second-best score among the 108 otolaryngologists who took the exam, while ChatGPT was placed 83rd. A chat powered by GPT-4 with internet access (Copilot) demonstrates superior performance in responding to multiple-choice medical questions compared to ChatGPT 3.5.
SUPPLEMENTAL MATERIAL
Coming soon ....
REFERENCES (15)
CITATIONS (6)
EXTERNAL LINKS
PlumX Metrics
RECOMMENDATIONS
FAIR ASSESSMENT
Coming soon ....
JUPYTER LAB
Coming soon ....