Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance

• DOI: 10.3389/fpubh.2024.1360597
• Publication Date: 2024-04-17T04:42:36Z
• Authors: Sultan Ayoub Meo; Metib Alotaibi; Muhammad Zain Sul...; Muhammad Omair Su...; Mashhood Hamid
• Keywords: knowledge; Health Knowledge, Attitudes, Practice; COVID-19 Vaccines; SARS-CoV-2; infectious disease; public health; COVID-19; 02 engineering and technology; artificial intelligence; Communicable Diseases; ChatGPT; Surveys and Questionnaires; 0202 electrical engineering, electronic engineering, information engineering; Humans; Public Health; Educational Measurement; Public aspects of medicine; RA1-1270; medical education; Pandemics

BackgroundAt the beginning of the year 2023, the Chatbot Generative Pre-Trained Transformer (ChatGPT) gained remarkable attention from the public. There is a great discussion about ChatGPT and its knowledge in medical sciences, however, literature is lacking to evaluate the ChatGPT knowledge level in public health. Therefore, this study investigates the knowledge of ChatGPT in public health, infectious diseases, the COVID-19 pandemic, and its vaccines.MethodsMultiple Choice Questions (MCQs) bank was established. The question’s contents were reviewed and confirmed that the questions were appropriate to the contents. The MCQs were based on the case scenario, with four sub-stems, with a single correct answer. From the MCQs bank, 60 MCQs we selected, 30 MCQs were from public health, and infectious diseases topics, 17 MCQs were from the COVID-19 pandemic, and 13 MCQs were on COVID-19 vaccines. Each MCQ was manually entered, and tasks were given to determine the knowledge level of ChatGPT on MCQs.ResultsOut of a total of 60 MCQs in public health, infectious diseases, the COVID-19 pandemic, and vaccines, ChatGPT attempted all the MCQs and obtained 17/30 (56.66%) marks in public health, infectious diseases, 15/17 (88.23%) in COVID-19, and 12/13 (92.30%) marks in COVID-19 vaccines MCQs, with an overall score of 44/60 (73.33%). The observed results of the correct answers in each section were significantly higher (p = 0.001). The ChatGPT obtained satisfactory grades in all three domains of public health, infectious diseases, and COVID-19 pandemic-allied examination.ConclusionChatGPT has satisfactory knowledge of public health, infectious diseases, the COVID-19 pandemic, and its vaccines. In future, ChatGPT may assist medical educators, academicians, and healthcare professionals in providing a better understanding of public health, infectious diseases, the COVID-19 pandemic, and vaccines.