A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis

• DOI: 10.1001/jamanetworkopen.2022.27423
• Publication Date: 2022-08-29T15:32:11Z
• Authors: Dongmei Sun; Thanh M. Nguyen; Robert J. Allaway; Jelai Wang; Verena Chung; Thomas V. Yu; Michael Mason; Isaac Dimitrovsky; Lars Ericson; Hongyang Li; Yuanfang Guan; Ariel Israel; Alex Olar; Balint Armin Pataki; Gustavo Stolovitzky; Justin Guinney; Percio S. Gulko; Mason B. Frazier; Jake Y. Chen; James C. Costello; S. Louis Bridges; ; Zbigniew Wojna; Anna Krason; YanMing Tan; RaphaelHaoChong Quek; Neelambuj Chaturvedi; Michael Stadler; Chenfu Shi; Krishnakumar Vait...; Julian Benadit; Duc Tran; Tin Nguyen; Alexander Biehl; Mehrad Mahmoudian; Sami Pietilä; Tomi Suomi; Mikko S Venäläinen; Laura L Elo; Chenguang Xue; Akshat Shreemali; Srinivas Chilukuri; Khanh-Tung Nguyen-Ba; Jay Ji-Hyung Ryu; Rui Bai; Yilin Wu; Yingnan Wu; Xiaofu He
• Keywords: Benchmark (surveying); Crowdsourcing

<h3>Importance</h3> An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images the hands wrists, feet clinical trials, monitoring damage over time, assisting rheumatologists with treatment decisions. Such a has potential to be directly integrated into electronic health records. <h3>Objectives</h3> To design implement an international crowdsourcing competition catalyze development machine learning methods quantify in rheumatoid arthritis (RA). <h3>Design, Setting, Participants</h3> This diagnostic/prognostic study describes Rheumatoid Arthritis 2–Dialogue Reverse Engineering Assessment Methods (RA2-DREAM Challenge), which used existing expert-curated Sharp-van der Heijde (SvH) scores from 2 studies (674 sets 562 patients) training (367 sets), leaderboard (119 final evaluation (188 sets). Challenge participants were tasked developing automatically overall (subchallenge 1), 2), 3). The challenge was finished June 30, 2020. <h3>Main Outcomes Measures</h3> Scores derived submitted algorithms compared SvH scores, baseline model created benchmark comparison. Performances ranked using weighted root mean square error (RMSE). performance reproductivity each algorithm assessed Bayes factor bootstrapped data, further evaluated postchallenge independent validation data set. <h3>Results</h3> RA2-DREAM received total 173 submissions 26 or teams 7 countries round, 13 included evaluation. RMSEs metric showed that winning produced very close scores. Top Team Shirin subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li Yuanfang Guan) 0.38), Gold Therapy 3 0.43). Bootstrapping/Bayes approach confirmed reproducibility estimation concordance indices between set 0.71 1, 0.78 2, 0.82 3. <h3>Conclusions Relevance</h3> resulted provide feasible, quick, RA. Ultimately, these could help research RA may records clinicians serve patients better by providing timely, reliable, quantitative information making decisions prevent damage.