R2-Bench: Benchmarking the Robustness of Referring Perception Models Under Perturbations
Li, Xiang Qiu, Kai Wang, Jinglu Xu, Xiaohao Singh, Rita Yamazaki, Kashu Chen, Hao Huang, Xiaonan Raj, Bhiksha
Li, Xiang
Qiu, Kai
Wang, Jinglu
Xu, Xiaohao
Singh, Rita
Yamazaki, Kashu
Chen, Hao
Huang, Xiaonan
Raj, Bhiksha
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Referring perception, which aims at grounding visual objects with multimodal referring guidance, is essential for bridging the gap between humans, who provide instructions, and the environment where intelligent systems perceive. Despite progress in this field, the robustness of referring perception models (RPMs) against disruptive perturbations is not well explored. This work thoroughly assesses the resilience of RPMs against various perturbations in both general and specific contexts. Recognizing the complex nature of referring perception tasks, we present a comprehensive taxonomy of perturbations, and then develop a versatile toolbox for synthesizing and evaluating the effects of composite disturbances. Employing this toolbox, we construct R2-Bench, a benchmark for assessing the Robustness of Referring perception models under noisy conditions across five key tasks. Moreover, we propose the R2-Agent, an LLM-based agent that simplifies and automates model evaluation via natural language instructions. Our investigation uncovers the vulnerabilities of current RPMs to various perturbations and provides tools for assessing model robustness, potentially promoting the safe and resilient integration of intelligent systems into complex real-world scenarios.
Citation
X. Li et al., “R 2 -Bench: Benchmarking the Robustness of Referring Perception Models Under Perturbations,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 15067 LNCS, pp. 211–230, 2025, doi: 10.1007/978-3-031-72673-6_12.
Source
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Conference
European Conference on Computer Vision (ECCV)
Keywords
Benchmark, Referring perception, Robustness & Perturbation
Subjects
Source
European Conference on Computer Vision (ECCV)
Publisher
Springer Nature