A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

• DOI: 10.1007/s10459-023-10225-y
• Publication Date: 2023-04-25T10:03:11Z
• Authors: Filipe Falcão; Daniela Marques P...; Nuno Gonçalves; Andre De Champlain; Patrício Costa; José Miguel Pêgo
• Keywords: Item bank; Classical test theory; Item analysis; Learnability

Abstract Automatic Item Generation (AIG) refers to the process of using cognitive models generate test items computer modules. It is a new but rapidly evolving research area where and psychometric theory are combined into digital framework. However, assessment item quality, usability validity AIG relative traditional development methods lacks clarification. This paper takes top-down strong approach evaluate in medical education. Two studies were conducted: Study I—participants with different levels clinical knowledge writing experience developed both manually through AIG. Both types compared terms quality (efficiency learnability) ; II—Automatically generated included summative exam content surgery. A analysis based on Response Theory inspected AIG-items. Items by presented evidences adequate for testing student’s knowledge. The time spent developing contents generation (cognitive models) number did not vary considering participants' or produces numerous high-quality fast, economical easy learn process, even inexperienced without training writers. Medical schools may benefit from substantial improvement cost-efficiency flaws can be significantly reduced thanks application AIG's models, thus generating capable accurately gauging students'