Mitigating object hallucination via concentric causal attention
Xing, Yun Li, Yiheng Laptev, Ivan Lu, Shijian
Xing, Yun
Li, Yiheng
Laptev, Ivan
Lu, Shijian
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recent Large Vision Language Models (LVLMs) present remarkable zero-shot conversational and reasoning capabilities given multimodal queries. Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs are prone to generate textual responses not factually aligned with image inputs. Our pilot study reveals that object hallucination is closely tied with Rotary Position Encoding (RoPE), a widely adopted positional dependency modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs tend to hallucinate more when relevant visual cues are distant from instruction tokens in the multimodal input sequence, Additionally, we observe a similar effect when reversing the sequential order of visual tokens during multimodal alignment. Our tests indicate that long-term decay in RoPE poses challenges to LVLMs while capturing visual-instruction interactions across long distances. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs by naturally reducing relative distance between visual and instruction tokens. With CCA, visual tokens can better interact with instruction tokens, thereby enhancing model's perception capability and alleviating object hallucination. Without bells and whistles, our positional alignment method surpasses existing hallucination mitigation strategies by large margins on multiple object hallucination benchmarks.
Citation
Y. Xing, Y. Li, I. Laptev, and S. Lu, “Mitigating Object Hallucination via Concentric Causal Attention,” Adv Neural Inf Process Syst, vol. 37, pp. 92012–92035, Dec. 2024, Accessed: Mar. 24, 2025. [Online]. Available: https://github.com/xing0047/cca-llava.git
Source
Advances in Neural Information Processing Systems (NeurIPS 2024)
Conference
Keywords
Object hallucination, Concentric Causal Attention (CCA), Rotary Position Encoding (RoPE), Large Vision-Language Models (LVLMs), Positional alignment
Subjects
Source
Publisher
NEURIPS