COMPKE: Complex Question Answering under Knowledge Editing
Cheng, Keyuan Kan, Zijian Zhang, Zhuoran Ali, Muhammad Asif Hu, Lijie Wang, Di
Cheng, Keyuan
Kan, Zijian
Zhang, Zhuoran
Ali, Muhammad Asif
Hu, Lijie
Wang, Di
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Knowledge Editing-Efficiently modifying the knowledge in large language models has gathered great attention. Current benchmarks primarily use multi-hop question answering to assess and analyze newly injected or updated knowledge. However, we argue that these benchmarks fail to effectively evaluate how well the updated models apply this knowledge in real-life scenarios, particularly when questions require complex reasoning involving one-to-many relationships or multi-step logical intersections. To fill in this gap, we introduce a new benchmark, COMPKE: Complex Question Answering under Knowledge Editing, which includes 11,924 complex questions that reflect real-life situations. We perform a comprehensive evaluation of four different knowledge editing methods in COMPKE, and our results show that the performance of these methods varies between different models. For example, MeLLo achieves an accuracy of 39.47 on GPT-4o-mini but drops significantly to 3.83 on Qwen2.5-3B. We further analyze the reasons behind these results from both methodological and model perspectives. Our dataset will be publicly available on GitHub.
Citation
K. Cheng, Z. Kan, Z. Zhang, M. A. Ali, L. Hu, and D. Wang, “COMPKE: Complex Question Answering under Knowledge Editing,” in Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, 2025, pp. 2557–2576.
Source
Findings of the Association for Computational Linguistics: NAACL 2025
Conference
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Keywords
Subjects
Source
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Publisher
Association for Computational Linguistics