PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

• DOI: 10.48550/arxiv.2405.10254
• Publication Date: 2024-05-16
• Authors: George Shaikovski; Adam Casson; Kristen Severson; Eric Zimmermann; Yi Kan Wang; Jeremy D. Kunz; Juan Antonio Retá...; Gerard J. Oakley; David S. Klimstra; Christopher Kanan; Matthew G. Hanna; Michal Zelechowski; Julian Viret; Neil Tenenholtz; James M. Hall; Nicolò Fusi; Razik Yousfi; Peter Hamilton; William A. Moye; Eugene Vorontsov; Siqi Liu; Thomas J. Fuchs
• Keywords: Prism; Foundation (evidence)

Foundation models in computational pathology promise to unlock the development of new clinical decision support systems and for precision medicine. However, there is a mismatch between most analysis, which defined at level one or more whole slide images, foundation date, process thousands image tiles contained separately. The requirement train network aggregate information across large number multiple images limits these models' impact. In this work, we present slide-level model H&E-stained histopathology, PRISM, that builds on Virchow tile embeddings leverages report text pre-training. Using embeddings, PRISM produces with ability generate reports, resulting several modes use. prompts, achieves zero-shot cancer detection sub-typing performance approaching surpassing supervised aggregator model. linear classifiers, surpasses models. Furthermore, demonstrate fine-tuning encoder yields label-efficient training biomarker prediction, task typically suffers from low availability data; an initialized trained as little 10% data can outperform baseline uses all data.