Inter- and Intra-Observer Variability and the Effect of Experience in Cine-MRI for Adhesion Detection

• DOI: 10.3390/jimaging9030055
• Publication Date: 2023-02-23T08:25:56Z
• Authors: Bram de Wilde; Frank Joosten; Wulphert Venderink; Mirjam E. J. Davidse; Juliëtte Geurts; Hanneke Kruijt; Afke Vermeulen; Bibi Martens; Maxime V. P. Schyns; Josephine C. B. M...; Myrte C. de Boer; Bart A. R. Tonino; Herman J. A. Zand...; Kirsti Lammert; Helka Parviainen; Aino-Maija Vuorinen; Suvi Syväranta; Ruben R. M. Vogels; Wiesje Prins; Andrea Coppola; Nancy Bossa; Richard P. G. ten...; Henkjan Huisman
• Keywords: tissue adhesions; Computer applications to medicine. Medical informatics; Radboudumc 10: Reconstructive and regenerative medicine Surgery; R858-859.7; Radboud University Medical Center; QA75.5-76.95; Radboudumc 15: Urological cancers Medical Imaging; CHRONIC ABDOMINAL-PAIN; observer variation; Article; 03 medical and health sciences; 0302 clinical medicine; Electronic computers. Computer science; Photography; (MeSH); magnetic resonance imaging; Radboudumc 14: Tumours of the digestive tract Surgery; INTRAABDOMINAL ADHESIONS; TR1-1050

Cine-MRI for adhesion detection is a promising novel modality that can help the large group of patients developing pain after abdominal surgery. Few studies into its diagnostic accuracy are available, and none address observer variability. This retrospective study explores the inter- and intra-observer variability, diagnostic accuracy, and the effect of experience. A total of 15 observers with a variety of experience reviewed 61 sagittal cine-MRI slices, placing box annotations with a confidence score at locations suspect for adhesions. Five observers reviewed the slices again one year later. Inter- and intra-observer variability are quantified using Fleiss’ (inter) and Cohen’s (intra) κ and percentage agreement. Diagnostic accuracy is quantified with receiver operating characteristic (ROC) analysis based on a consensus standard. Inter-observer Fleiss’ κ values range from 0.04 to 0.34, showing poor to fair agreement. High general and cine-MRI experience led to significantly (p < 0.001) better agreement among observers. The intra-observer results show Cohen’s κ values between 0.37 and 0.53 for all observers, except one with a low κ of −0.11. Group AUC scores lie between 0.66 and 0.72, with individual observers reaching 0.78. This study confirms that cine-MRI can diagnose adhesions, with respect to a radiologist consensus panel and shows that experience improves reading cine-MRI. Observers without specific experience adapt to this modality quickly after a short online tutorial. Observer agreement is fair at best and area under the receiver operating characteristic curve (AUC) scores leave room for improvement. Consistently interpreting this novel modality needs further research, for instance, by developing reporting guidelines or artificial intelligence-based methods.