Admitting Ignorance Helps the Video Question Answering Models to Answer
Li, Haopeng Drummond, Tom Gong, Mingming Bennamoun, Mohammed Ke, Qiuhong
Li, Haopeng
Drummond, Tom
Gong, Mingming
Bennamoun, Mohammed
Ke, Qiuhong
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Significant progress has been made in the field of video question answering (VideoQA) thanks to deep learning and large-scale pretraining. Despite the presence of sophisticated model structures and powerful video-text foundation models, most existing methods focus solely on maximizing the correlation between answers and video-question pairs during training. We argue that these models often establish shortcuts, resulting in spurious correlations between questions and answers, especially when the alignment between video and text data is suboptimal. To address these spurious correlations, we propose a novel training framework in which the model is compelled to acknowledge its ignorance when presented with an intervened question, rather than making guesses solely based on superficial question-answer correlations. We introduce methodologies for intervening in questions, utilizing techniques such as displacement and perturbation, and design frameworks for the model to admit its lack of knowledge in both multi-choice VideoQA and open-ended settings. In practice, we integrate a state-of-the-art model into our framework to validate its effectiveness. The results clearly demonstrate that our framework can significantly enhance the performance of VideoQA models with minimal structural modifications.
Citation
H. Li, T. Drummond, M. Gong, M. Bennamoun, Q. Ke, "Admitting Ignorance Helps the Video Question Answering Models to Answer," IEEE Transactions on Multimedia, vol. PP, no. 99, pp. 1-13, 2026, https://doi.org/10.1109/tmm.2026.3651115.
Source
IEEE Transactions on Multimedia
Conference
Keywords
46 Information and Computing Sciences, 4605 Data Management and Data Science
Subjects
Source
Publisher
IEEE