MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems

• DOI: 10.48550/arxiv.2410.14179
• Publication Date: 2024-10-18
• Authors: Zifeng Zhu; Mengzhao Jia; Zhihan Zhang; Lang Li; Meng Jiang
• Keywords: Benchmarking

Multimodal Large Language Models (MLLMs) have demonstrated impressive abilities across various tasks, including visual question answering and chart comprehension, yet existing benchmarks for chart-related tasks fall short in capturing the complexity of real-world multi-chart scenarios. Current primarily focus on single-chart neglecting multi-hop reasoning required to extract integrate information from multiple charts, which is essential practical applications. To fill this gap, we introduce MultiChartQA, a benchmark that evaluates MLLMs' capabilities four key areas: direct answering, parallel comparative reasoning, sequential reasoning. Our evaluation wide range MLLMs reveals significant performance gaps compared humans. These results highlight challenges comprehension potential MultiChartQA drive advancements field. code data are available at https://github.com/Zivenzhu/Multi-chart-QA