Enhancement of the Performance of Large Language Models in Diabetes Education through Retrieval-Augmented Generation: Comparative Study

• DOI: 10.2196/58041
• Publication Date: 2024-07-24T11:38:23Z
• Authors: Dingqiao Wang; Jiangbo Liang; Jinguo Ye; Jingni Li; Jingpeng Li; Qikai Zhang; Qiuling Hu; Caineng Pan; Dongliang Wang; Zhong Liu; Wen Shi; Danli Shi; Fei Li; Bo Qu; Yingfeng Zheng
• Keywords: 0301 basic medicine; Original Paper; Diabetes education; Computer applications to medicine. Medical informatics; R858-859.7; Claude-2; RAG; GPT-4.0; Retrieval-augmented generation; Google Bard; 0302 clinical medicine; LLMs; Large language models; Public aspects of medicine; RA1-1270

Background Large language models (LLMs) demonstrated advanced performance in processing clinical information. However, commercially available LLMs lack specialized medical knowledge and remain susceptible to generating inaccurate Given the need for self-management diabetes, patients commonly seek information online. We introduce Retrieval-augmented Information System Enhancement (RISE) framework evaluate its enhancing provide accurate responses diabetes-related inquiries. Objective This study aimed potential of RISE framework, an retrieval augmentation tool, improve LLM’s accurately safely respond Methods The RISE, innovative comprises 4 steps: rewriting query, retrieval, summarization, execution. Using a set 43 common questions, we evaluated 3 base (GPT-4, Anthropic Claude 2, Google Bard) their RISE-enhanced versions respectively. Assessments were conducted by clinicians accuracy comprehensiveness understandability. Results integration significantly improved from all LLMs. On average, percentage increased 12% (15/129) with RISE. Specifically, rates 7% (3/43) GPT-4, 19% (8/43) 9% (4/43) Bard. also enhanced response comprehensiveness, mean scores improving 0.44 (SD 0.10). Understandability was 0.19 0.13) on average. Data collection September 30, 2023 February 5, 2024. Conclusions improves LLMs’ responding inquiries, accuracy, These improvements have crucial implications RISE’s future role patient education chronic illness self-management, which contributes relieving resource pressures raising public awareness knowledge.