Advancing Multimodal Medical Capabilities of Gemini

• DOI: 10.48550/arxiv.2405.03162
• Publication Date: 2024-05-06
• Authors: Lin Yang; Shawn Xu; Andrew Sellergren; Timo Kohlberger; Yuchen Zhou; Sofia Ira Ktena; Atilla P. Kiraly; Faruk Ahmed; Farhad Hormozdiari; Tiam Jaroensri; E.-W. Wang; Ellery Wulczyn; F Guimaraes Silvi...; Theo Guidroz; Chuck Lau; Siyuan Qiao; Yun Liu; Akshay Goel; Kendall Park; Arnav Agharwal; Nick George; Yang Wang; Ryutaro Tanno; David G. T. Barrett; Wei‐Hung Weng; S. Sara Mahdavi; Khaled Saab; Tao Tu; Sreenivasa Raju K...; Mozziyar Etemadi; Jorge Cuadros; Gregory Sorensen; Yossi Matias; Katherine Chou; G. Corrado; Joëlle Barral; Shravya Shetty; David J. Fleet; S. M. Ali Eslami; Daniel Tse; Shruthi Prabhakara; Cory Y. McLean; Dave Steiner; Rory Pilgrim; Christopher Kelly; Shekoofeh Azizi; Daniel Golden
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; Computation and Language (cs.CL); Machine Learning (cs.LG)

Many clinical tasks require an understanding of specialized data, such as medical images and genomics, which is not typically found in general-purpose large multimodal models. Building upon Gemini's models, we develop several models within the new Med-Gemini family that inherit core capabilities Gemini are optimized for use via fine-tuning with 2D 3D radiology, histopathology, ophthalmology, dermatology genomic data. Med-Gemini-2D sets a standard AI-based chest X-ray (CXR) report generation based on expert evaluation, exceeding previous best results across two separate datasets by absolute margin 1% 12%, where 57% 96% AI reports normal cases, 43% 65% abnormal evaluated "equivalent or better" than original radiologists' reports. We demonstrate first ever model-based computed tomography (CT) volumes using Med-Gemini-3D, 53% considered clinically acceptable, although additional research needed to meet radiologist reporting quality. Beyond generation, surpasses performance CXR visual question answering (VQA) performs well classification radiology VQA, SoTA baselines 17 20 tasks. In image classification, 18 out approaches task-specific model performance. imaging, Med-Gemini-Polygenic outperforms linear polygenic risk score-based approach disease prediction generalizes genetically correlated diseases it has never been trained. Although further development evaluation necessary safety-critical domain, our highlight potential wide range