OSAIRIS: Lessons Learned From the Hospital-Based Implementation and Evaluation of an Open-Source Deep-Learning Model for Radiotherapy Image Segmentation

• DOI: 10.1016/j.clon.2024.10.032
• Publication Date: 2024-10-18T23:03:33Z
• Authors: A.D. Constantinou; A. Hoole; D.C. Wong; G.S. Sagoo; J. Alvarez-Valle; K. Takeda; T. Griffiths; A. Edwards; A. Robinson; L. Stubbington; N. Bolger; Y. Rimmer; T. Elumalai; K.T. Jayaprakash; R. Benson; I. Gleeson; R. Sen; L. Stockton; T. Wang; S. Brown; E. Gatfield; C. Sanghera; A. Mourounas; B. Evans; A. Anthony; R. Hou; M. Toomey; K. Wildschut; A. Grisby; G.C. Barnett; R. McMullen; R. Jena
• Keywords: Radiotherapy; Accuracy metrics; name=Oncology; Human-Centric AI; 610; Artificial Intelligence (AI); name=Cancer Research; Autosegmentation; name=Artificial Intelligence; Machine Learning; /dk/atira/pure/subjectarea/asjc/2700/2741; /dk/atira/pure/subjectarea/asjc/2700/2730; Accuracy assessment; AI; name=Radiology Nuclear Medicine and imaging; 616; Medical Device; Human-centric; /dk/atira/pure/subjectarea/asjc/1300/1306; Machine-learning; /dk/atira/pure/subjectarea/asjc/1700/1702; Cancer

Several studies report the benefits and accuracy of using autosegmentation for organ at risk (OAR) outlining in radiotherapy treatment planning. Typically, evaluations focus on accuracy metrics, and other parameters such as perceived utility and safety are routinely ignored. Here we report our finding from the implementation and clinical evaluation of OSAIRIS, an open-source AI model for radiotherapy image segmentation, that was carried out as part of its development into a medical device. The device contours OARs in the head and neck and male pelvis (referred to as the prostate model), and is designed to be used as a time-saving workflow device, alongside a clinician. Unlike standard evaluation processes, which heavily rely on accuracy metrics alone, our evaluation sought to demonstrate the tangible benefits, quantify utility and assess risk within a specific clinical workflow. We evaluated the time-saving benefit this device affords to clinicians, and how this time-saving might be linked to accuracy metrics, as well as the clinicians’ assessment of the usability of the OSAIRIS contours in comparison to their colleagues’ contours and those from other commercial AI contouring devices. Our safety evaluation focused on whether clinicians can notice and correct any errors should they be included in the output of the device. We found that OSAIRIS affords a significant time-saving of 36% (5.4 ± 2.1 minutes) when used for prostate contouring and 67% (30.3 ± 8.7 minutes) for head and neck contouring. Combining editing time data with accuracy metrics, we found the Hausdorff distance best correlated with editing-time, outperforming dice, the industry-standard, with a Spearman correlation coefficient of 0.70, and a Kendall coefficient of 0.52. Our safety and risk-mitigation exercise showed that anchoring bias is present when clinicians edit AI-generated contours, with the effect seemingly more pronounced for some structures over others. Most errors, however, were corrected by clinicians, with 72% of the head and neck ...