Generating learning guides for medical education with LLMs and statistical analysis of test results

• DOI: 10.1186/s12909-025-06978-2
• Publication Date: 2025-03-30T02:10:31Z
• Authors: Iván Roselló Atanet; Mihaela Tomova; Miriam Sieg; Victoria Sehy; Patrick Mäder; Maren März
• Keywords: Strengths and weaknesses

Abstract Background The Progress Test Medizin (PTM) is a formative test for medical students issued twice year by the Charité-Universitätsmedizin Berlin. PTM provides numerical feedback based on global view of strengths and weaknesses students. This can benefit from more fine-grained information, pinpointing topics where need to improve, as well advice what they should learn in light their results. scale PTM, taken than 10,000 participants every academic semester, makes it necessary automate this task. Methods We have developed seven-step approach large language models statistical analysis fulfil purpose study. Firstly, model (ChatGPT 4.0) identified keywords form MeSH terms all 200 questions one run. These were checked against list included Medical Subject Headings (MeSH) thesaurus published National Library Medicine (NLM). Meanwhile, answer patterns also analysed find empirical relationships between questions. With we obtained series related specific used them develop framework that allowed us assess performance compose personalized structured around curated topics. Results data past simulate generation 1,401 participants, thereby producing information about knowledge regarding number ranging 34 243. Substantial gaps found 14.67% 21.76% rated learning topics, depending benchmarking set considered. Conclusion designed tested method generate student covering up 243 defined terms. generated with later stages studies was detailed, tend face matching level.