The Diagnostic Ability of GPT-3.5 and GPT-4.0 in Surgery: Comparative Analysis (Preprint)

• DOI: 10.2196/preprints.54985
• Publication Date: 2024-09-10T14:16:08Z
• Authors: Jiayu Liu; Xiuting Liang; Dandong Fang; Jiqi Zheng; Chengliang Yin; Hui Xie; Yanteng Li; Xiaochun Sun; Yue Tong; Hebin Che; Ping Hu; Fan Yang; Bingxian Wang; Yuanyuan Chen; Gang Cheng; Jianning Zhang
• Keywords: Preprint

<sec> <title>BACKGROUND</title> ChatGPT (OpenAI) has shown great potential in clinical diagnosis and could become an excellent auxiliary tool practice. This study investigates evaluates diagnostic capabilities by comparing the performance of GPT-3.5 GPT-4.0 across model iterations. </sec> <title>OBJECTIVE</title> aims to evaluate precise ability for colon cancer its as surgeons compare accuracy rates between GTP-3.5 GPT-4.0. We precisely assess primary secondary diagnoses analyze causes misdiagnoses according 7 categories: patient histories, symptoms, physical signs, laboratory examinations, imaging pathological intraoperative findings. <title>METHODS</title> retrieved 316 case reports intestinal from Chinese Medical Association Publishing House database, which 286 cases were deemed valid after data cleansing. The translated Mandarin English then input into using a simple, direct prompt elicit diagnoses. conducted comparative GPT-3.5. Three senior General Surgery Department, specializing Colorectal Surgery, assessed information at PLA (People’s Liberation Army) Hospital. was scored based on predefined criteria. Additionally, we analyzed compared both models <title>RESULTS</title> Out cases, demonstrated high diagnoses, but significantly higher than (mean 0.972, SD 0.137 vs mean 0.855, 0.335; &lt;i&gt;t&lt;/i&gt;&lt;sub&gt;285&lt;/sub&gt;=5.753; &lt;i&gt;P&lt;/i&gt;&amp;lt;.001). For also 0.908, 0.159 0.617, 0.349; &lt;i&gt;t&lt;/i&gt;&lt;sub&gt;285&lt;/sub&gt;=–7.727; showed limitations processing history, symptom presentation, tests, data. While improved upon GPT-3.5, it still identifying symptoms test there no significant difference related age, gender, or system group <title>CONCLUSIONS</title> demonstrates that ChatGPT, particularly GPT-4.0, possesses potential, with exhibiting However, limitations, recognizing data, indicating need more research real-world settings enhance capabilities.