Reliability of a generative artificial intelligence tool for pediatric familial Mediterranean fever: insights from a multicentre expert survey

• DOI: 10.1186/s12969-024-01011-0
• Publication Date: 2024-08-23T22:15:53Z
• Authors: Saverio La Bella; Marina Attanasi; Annamaria Porreca; Armando Di Ludovico; Maria Cristina Ma...; Romina Gallizzi; Francesco La Torre; Donato Rigante; Francesca Soscia; Francesca Ardenti...; Antonella Insalaco; Marco Francesco N...; Francesco Chiarelli; Gabriele Simonini; Fabrizio De Bened...; Marco Gattorno; Luciana Breda
• Keywords: Artificial intelligence; Diseases of the musculoskeletal system; Familial Mediterranean fever; Pediatrics; RJ1-570; 03 medical and health sciences; Settore MED/38 - Pediatria Generale E Specialistica; FMF; 0302 clinical medicine; AI; Artificial intelligence; FMF; Familial mediterranean fever; Generative artificial intelligence; Pediatric rheumatology; Artificial Intelligence; Surveys and Questionnaires; Humans; Pediatric rheumatology; Generative artificial intelligence; Child; Observer Variation; Reproducibility of Results; Familial Mediterranean Fever; RC925-935; AI; Familial mediterranean fever; AI, Artificial intelligence, FMF, Familial mediterranean fever, Generative artificial intelligence, Pediatric rheumatology; Research Article

Artificial intelligence (AI) has become a popular tool for clinical and research use in the medical field. The aim of this study was to evaluate accuracy reliability generative AI on pediatric familial Mediterranean fever (FMF). Fifteen questions repeated thrice FMF were prompted Microsoft Copilot with Chat-GPT 4.0. Nine rheumatology experts rated response blinded mechanism using Likert-like scale values from 1 5. Median overall responses at initial assessment ranged 2.00 5.00. During second assessment, median spanned 4.00, while third they 3.00 4.00. Intra-rater variability showed poor moderate agreement (intraclass correlation coefficient range: -0.151 0.534). A diminishing level among over time documented, as highlighted by Krippendorff's alpha values, ranging 0.136 (at first response) 0.132 0.089 response). Lastly, displayed varying levels trust pre- post-survey. promising implications rheumatology, including early diagnosis management optimization, but challenges persist due uncertain information lack expert validation. Our survey revealed considerable inaccuracies incompleteness AI-generated regarding FMF, intra- extra-rater reliability. Human validation remains crucial managing information.