Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening

• DOI: 10.1056/aioa2400181
• Publication Date: 2024-06-17T14:01:01Z
• Authors: Ozan Unlu; Jiyeon Shin; Charlotte J. Mailly; Michael F. Oates; Michela R. Tucci; Matthew Varugheese; Kavishwar Wagholikar; Fei Wang; Benjamin M. Scirica; Alexander J. Blood; Samuel J. Aronson

BackgroundScreening participants in clinical trials is an error-prone and labor-intensive process that requires significant time resources. Large language models such as generative pretrained transformer 4 (GPT-4) present opportunity to enhance the screening with advanced natural processing. This study evaluates utility of a Retrieval-Augmented Generation (RAG)–enabled GPT-4 system improve accuracy, efficiency, reliability for trial involving patients symptomatic heart failure.MethodsThe ongoing Co-Operative Program Implementation Optimal Therapy Heart Failure (COPILOT-HF; ClinicalTrials.gov number, NCT05734690) identifies potential through electronic health record (EHR) queries followed by manual reviews trained but nonlicensed staff. To determine patient eligibility COPILOT-HF not identifiable structured EHR queries, we developed RAG-Enabled Clinical Trial Infrastructure Inclusion Exclusion Review (RECTIFIER), note–based, question-answering powered RAG GPT-4. We used notes on 100, 282, 1894 development, validation, test datasets, respectively. An expert clinician conducted blinded review establish "gold standard" answers 13 target criteria questions. calculated performance metrics (sensitivity, specificity, Matthews correlation coefficient [MCC]) determining each criterion four methods (study staff, RECTIFIER single-question strategy, combined-question GPT-3.5 instead GPT-4).ResultsThe staff's closely aligned clinicians' across criteria, accuracy ranging between 97.9% 100% (MCC, 0.837 1) 91.7% 0.644 performed better than staff failure, versus MCC 0.924 0.721, Overall, sensitivity specificity were 92.3% 93.9%, respectively, 90.1% 83.6% With RECTIFIER, approach resulted average cost 11 cents per patient, 2 patient.ConclusionsLarge model–based solutions can significantly reduce costs automating process. However, integrating technologies careful consideration hazards should include safeguards final review. (Funded Accelerator Transformation [ACT]; NCT05734690.)