Loading...
Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue
Purkayastha, Sukannya Dycke, Nils Lauscher, Anne Gurevych, Iryna
Purkayastha, Sukannya
Dycke, Nils
Lauscher, Anne
Gurevych, Iryna
Files
Loading...
2026.eacl-long.158.pdf
Adobe PDF, 5.39 MB
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
License
http://creativecommons.org/licenses/by/4.0/
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Meta-reviewing is a pivotal stage in the peer-review process, serving as the final step in determining whether a paper is recommended for acceptance. Prior research on meta-reviewing has treated this as a summarization problem over review reports. However, complementary to this perspective, meta-reviewing is a decision-making process that requires weighing reviewer arguments and placing them within a broader context. Prior research has demonstrated that decision-makers can be effectively assisted in such scenarios via dialogue agents. In line with this framing, we explore the practical challenges for realizing dialog agents that can effectively assist meta-reviewers. Concretely, we first address the issue of data scarcity for training dialogue agents by generating synthetic data using Large Language Models (LLMs) based on a self-refinement strategy to improve the relevance of these dialogues to expert domains. Our experiments demonstrate that this method produces higher-quality synthetic data and can serve as a valuable resource towards training meta-reviewing assistants. Subsequently, we utilize this data to train dialogue agents tailored for meta-reviewing and find that these agents outperform *off-the-shelf* LLM-based assistants for this task. Finally, we apply our agents in real-world meta-reviewing scenarios and confirm their effectiveness in enhancing the efficiency of meta-reviewing.
Citation
S. Purkayastha, N. Dycke, A. Lauscher, I. Gurevych, "Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue," 2026, pp. 3424-3465.
Source
Conference
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Keywords
Subjects
Source
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics