Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation

• DOI: 10.48550/arxiv.2501.03225
• Publication Date: 2025-01-01
• Authors: Zhang, Yuhui; Su, Yuchang; Liu, Yiming; Wang, Xiaohan; Burgess, James; Sui, Elaine; Wang, Chenyu; Aklilu, Josiah; Lozano, Alejandro; Wei, Anjiang; Schmidt, Ludwig; Yeung-Levy, Serena
• Keywords: FOS: Computer and information sciences; Computer Science - Computers and Society; Computer Science - Machine Learning; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computer Vision and Pattern Recognition (cs.CV); Computers and Society (cs.CY); Computer Science - Computer Vision and Pattern Recognition; Computation and Language (cs.CL); Machine Learning (cs.LG)

The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation. However, current visual question answering (VQA) benchmarks often depend on open-ended questions, making accurate evaluation difficult due to the variability in natural language responses. To address this, we introduce AutoConverter, an agentic framework that automatically converts these open-ended questions into multiple-choice format, enabling objective evaluation while reducing the costly question creation process. Our experiments demonstrate that AutoConverter can generate correct and challenging multiple-choice questions, with VLMs demonstrating consistently similar or lower accuracy on these questions compared to human-created ones. Using AutoConverter, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format, totaling 9,018 questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.<br/>Project page: https://yuhui-zh15.github.io/AutoConverter-Website/<br/>