Loading...
CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
Ghaboura, Sara Heakl, Ahmed Thawakar, Omkar Alharthi, Ali Husain Salem Abdulla Riahi, Ines Radman, Abduljalil Laaksonen, Jorma Khan, Fahad Shahbaz Khan, Salman Anwer, Rao Muhammad
Ghaboura, Sara
Heakl, Ahmed
Thawakar, Omkar
Alharthi, Ali Husain Salem Abdulla
Riahi, Ines
Radman, Abduljalil
Laaksonen, Jorma
Khan, Fahad Shahbaz
Khan, Salman
Anwer, Rao Muhammad
Files
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
License
http://creativecommons.org/licenses/by/4.0/
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recent years have witnessed a significant interest in developing large multi-modal models (LMMs) capable of performing various visual reasoning and understanding tasks. This has led to the introduction of multiple LMM benchmarks to evaluate LMMs on different tasks. However, most existing LMM evaluation benchmarks are predominantly English-centric. In this work, we develop a comprehensive LMM evaluation benchmark for the Arabic language to represent a large population of over 400 million speakers. The proposed benchmark, named CAMEL-Bench, comprises eight diverse domains and 38 sub-domains including, multi-image understanding, complex visual perception, handwritten document understanding, video understanding, medical imaging, plant diseases, and remote sensing-based land use understanding to evaluate broad scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions that are filtered from a larger pool of samples, where the quality is manually verified by native speakers to ensure reliable model assessment. We conduct evaluations of both closed-source, including GPT-4 series, and open-source LMMs. Our analysis reveals the need for substantial improvement, especially among the bestopen-source models, with even the closed-source GPT-4o achieving an overall score of 62%. Our benchmark will be publicly released.
Citation
S. Ghaboura, A. Heakl, O. Thawakar, A.H.S.A. Alharthi, I. Riahi, A. Radman, J. Laaksonen, F.S. Khan, S. Khan, R.M. Anwer, "CAMEL-Bench: A Comprehensive Arabic LMM Benchmark," 2025, pp. 1970-1980.
Source
Proceedings of the Conference Findings, NAACL 2025
Conference
2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Keywords
Subjects
Source
2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Publisher
Association for Computational Linguistics