Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

• DOI: 10.48550/arxiv.2310.09909
• Publication Date: 2023-01-01
• Authors: Chaoyi Wu; Jiayu Lei; Qiaoyu Zheng; Weike Zhao; Weixiong Lin; Xiaoman Zhang; Xiao Zhou; Ziheng Zhao; Ya Zhang; Yanfeng Wang; Weidi Xie
• Keywords: Digital subtraction angiography; Modalities; Modality (human–computer interaction)

Driven by the large foundation models, development of artificial intelligence has witnessed tremendous progress lately, leading to a surge general interest from public. In this study, we aim assess performance OpenAI's newest model, GPT-4V(ision), specifically in realm multimodal medical diagnosis. Our evaluation encompasses 17 human body systems, including Central Nervous System, Head and Neck, Cardiac, Chest, Hematology, Hepatobiliary, Gastrointestinal, Urogenital, Gynecology, Obstetrics, Breast, Musculoskeletal, Spine, Vascular, Oncology, Trauma, Pediatrics, with images taken 8 modalities used daily clinic routine, e.g., X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission (PET), Digital Subtraction Angiography (DSA), Mammography, Ultrasound, Pathology. We probe GPT-4V's ability on multiple clinical tasks or without patent history provided, imaging modality anatomy recognition, disease diagnosis, report generation, localisation. observation shows that, while GPT-4V demonstrates proficiency distinguishing between image anatomy, it faces significant challenges diagnosis generating comprehensive reports. These findings underscore that models have made advancements computer vision natural language processing, remains far being effectively support real-world applications decision-making. All can be found https://github.com/chaoyi-wu/GPT-4V_Medical_Evaluation.