MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

• DOI: 10.48550/arxiv.2110.01406
• Publication Date: 2021-01-01
• Authors: Alexandros Kararg...; Renato Umeton; Micah Sheller; Alejandro Aristiz...; Johnu George; Srini Bala; Daniel J. Beutel; Victor Bittorf; Akshay Chaudhari; Alexander Chowdhury; Cody Coleman; Bala Desinghu; Gregory Diamos; Debo Dutta; Diane Feddema; Grigori Fursin; Junyi Guo; Xinyuan Huang; David Kanter; Satyananda Kashyap; Nicholas D. Lane; Indranil Mallick; Pietro Mascagni; Virendra Mehta; Vivek Natarajan; Nikola S. Nikolov; Nicolas Padoy; Gennady Pekhimenko; Vijay Janapa Reddi; G. Anthony Reina; Pablo Ribalta; Jacob Rosenthal; Abhishek Singh; Jayaraman J. Thia...; Anna Wuest; Maria Xenochristou; Daguang Xu; Poonam Yadav; Michael H. Rosenthal; Massimo Loda; Jason M. Johnson; Peter Mattson
• Keywords: Benchmarking

Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider experience. We argue that unlocking this requires a systematic way measure performance medical models on large-scale heterogeneous data. To meet need, we are building MedPerf, an open framework for benchmarking machine learning in domain. MedPerf will enable federated evaluation which securely distributed different facilities evaluation, thereby empowering organizations assess verify efficient human-supervised process, while prioritizing privacy. describe current challenges communities face, need platform, design philosophy its implementation status, our roadmap. call researchers join us creating platform.