Loading...
ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
Ghaboura, Sara Patle, Shubham More, Ketan Alghallabi, Wafa Hamad Mohamed Thawakar, Omkar Laaksonen, Jorma Cholakkal, Hisham Khan, Salman Anwer, Rao
Ghaboura, Sara
Patle, Shubham
More, Ketan
Alghallabi, Wafa Hamad Mohamed
Thawakar, Omkar
Laaksonen, Jorma
Cholakkal, Hisham
Khan, Salman
Anwer, Rao
Files
Loading...
2026.lrec2026-1.723.pdf
Adobe PDF, 3.41 MB
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
License
http://creativecommons.org/licenses/by/4.0/
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
As Large Multimodal Models (LMMs) become more capable, there is growing interest in evaluating their reasoning processes alongside their final outputs. However, most existing benchmarks remain focused on English, overlooking languages with rich linguistic and cultural depth such as Arabic. To address this gap, we introduce the Comprehensive Arabic Multimodal Reasoning Benchmark (ARB), the first benchmark designed to evaluate step-by-step reasoning in Arabic across both textual and visual modalities. ARB covers 11 diverse domains and over 40 subfields, including visual reasoning, optical character recognition, scientific analysis, and cultural interpretation. It comprises 2,219 multimodal samples paired with over 8K human-curated reasoning steps and corresponding actions, verified through a human-in-the-loop process. We evaluated 15 state-of-the-art open- and closed-source LMMs and found persistent challenges in coherence, faithfulness, and cultural grounding. ARB provides a structured framework for diagnosing multimodal reasoning in underrepresented languages, marking a critical step toward inclusive, transparent, and culturally aware AI systems. The benchmark, rubric, and evaluation suite are publicly available
Citation
S. Ghaboura, S. Patle, K. More, W.H.M. Alghallabi, O. Thawakar, J. Laaksonen , et al., "ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark," 2026, pp. 9202-9216.
Source
Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)
Conference
The Fifteenth Language Resources and Evaluation Conference (LREC 2026)
Keywords
Subjects
Source
The Fifteenth Language Resources and Evaluation Conference (LREC 2026)
Publisher
ELDA (Evaluations and Language resources Distribution Agency)