Peer-Reviewed Publication
Nat Mach Intell2023;5(7):799-810.July 1, 2023Journal Article
Federated benchmarking of medical artificial intelligence with MedPerf.
Alexandros Karargyris1,2,3, Renato Umeton4,5,6,7,3, Micah J Sheller8,3, Alejandro Aristizabal9, Johnu George10, Anna Wuest4,6, Sarthak Pati11,12, Hasan Kassem2, Maximilian Zenk13,14, Ujjwal Baid11,12, Prakash Narayana Moorthy8, Alexander Chowdhury4, Junyi Guo6, Sahil Nalawade4, Jacob Rosenthal4,5, David Kanter15, Maria Xenochristou16, Daniel J Beutel17,18, Verena Chung19, Timothy Bergquist19, James Eddy19, Abubakar Abid20, Lewis Tunstall20, Omar Sanseviero20, Dimitrios Dimitriadis21, Yiming Qian22, Xinxing Xu22, Yong Liu22, Rick Siow Mong Goh22, Srini Bala23, Victor Bittorf24, Sreekar Reddy Puchala4, Biagio Ricciuti4, Soujanya Samineni4, Eshna Sengupta4, Akshay Chaudhari16,25, Cody Coleman16, Bala Desinghu26, Gregory Diamos27, Debo Dutta10, Diane Feddema28, Grigori Fursin29,30, Xinyuan Huang31, Satyananda Kashyap32, Nicholas Lane17,18, Indranil Mallick33, , , , Pietro Mascagni1,34, Virendra Mehta35, Cassiano Ferro Moraes36, Vivek Natarajan37, Nikola Nikolov23, Nicolas Padoy1,2, Gennady Pekhimenko38,39, Vijay Janapa Reddi40, G Anthony Reina8, Pablo Ribalta41, Abhishek Singh7, Jayaraman J Thiagarajan42, Jacob Albrecht19, Thomas Wolf20, Geralyn Miller21, Huazhu Fu22, Prashant Shah8, Daguang Xu41, Poonam Yadav43, David Talby44, Mark M Awad4,45, Jeremy P Howard46,47, Michael Rosenthal4,45,48, Luigi Marchionni5, Massimo Loda4,5,45,49, Jason M Johnson4, Spyridon Bakas11,12,50, Peter Mattson15,37,50
1IHU Strasbourg, Strasbourg, France.
2University of Strasbourg, Strasbourg, France.
3These authors contributed equally: Alexandros Karargyris, Renato Umeton, Micah J. Sheller.
4Dana-Farber Cancer Institute, Boston, MA, USA.
5Weill Cornell Medicine, New York, NY, USA.
6Harvard T.H. Chan School of Public Health, Boston, MA, USA.
7Massachusetts Institute of Technology, Cambridge, MA, USA.
8Intel, Santa Clara, CA, USA.
9Factored, Palo Alto, CA, USA.
10Nutanix, San Jose, CA, USA.
11Perelman School of Medicine, Philadelphia, PA, USA.
12University of Pennsylvania, Philadelphia, PA, USA.
13German Cancer Research Center, Heidelberg, Germany.
14University of Heidelberg, Heidelberg, Germany.
15MLCommons, San Francisco, CA, USA.
16Stanford University, Stanford, CA, USA.
17University of Cambridge, Cambridge, UK.
18Flower Labs, Hamburg, Germany.
19Sage Bionetworks, Seattle, WA, USA.
20Hugging Face, New York, NY, USA.
21Microsoft, Redmond, WA, USA.
22A*STAR, Singapore, Singapore.
23Supermicro, San Jose, CA, USA.
24Meta, Menlo Park, CA, USA.
25Stanford University School of Medicine, Stanford, CA, USA.
26Rutgers University, New Brunswick, NJ, USA.
27Landing.AI, Palo Alto, CA, USA.
28Red Hat, Raleigh, NC, USA.
29cKnowledge, Paris, France.
30OctoML, Seattle, WA, USA.
31Cisco, San Jose, CA, USA.
32IBM Research, San Jose, CA, USA.
33Tata Medical Center, Kolkata, India.
34Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy.
35University of Trento, Trento, Italy.
36Write Choice, Florianópolis, Brazil.
37Google, Mountain View, CA, USA.
38University of Toronto, Toronto, Ontario, Canada.
39Vector Institute, Toronto, Ontario, Canada.
40Harvard University, Cambridge, MA, USA.
41NVIDIA, Santa Clara, CA, USA.
42Lawrence Livermore National Laboratory, Livermore, CA, USA.
43University of York, York, UK.
44John Snow Labs, Lewes, DE, USA.
45Harvard Medical School, Boston, MA, USA.
46fast.ai, San Francisco, CA, USA.
47University of Queensland, Brisbane, Queensland, Australia.
48Brigham and Women's Hospital, Boston, MA, USA.
49Broad Institute of MIT and Harvard, Cambridge, MA, USA.
50These authors jointly supervised this work: Spyridon Bakas, Peter Mattson.
Abstract
Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, het…
Create a free account to keep reading
Free members get 10 full research views every month across publications, clinical trials, FDA clearances, adverse events, and NIH grants. No credit card required.
Want unlimited research access? See Pro plans