Exploring the Integration of Large Language Models in Peer Review: Challenges, Opportunities, and Implications
Afzal, Osama Mohammed
Afzal, Osama Mohammed
Author
Supervisor
Department
Natural Language Processing
Embargo End Date
01/01/2024
Type
Thesis
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The increasing accessibility of large language models (LLMs) has raised concerns about their potential impact on the peer review process in scientific research. This study explores the challenges and opportunities associated with integrating LLMs into the peer review ecosystem. We establish a framework to examine two scenarios: (1) reviewers using AI for generating entire reviews, and (2) reviewers collaborating with AI in a mixed-authorship setting. Our refined problem formulation focuses on three primary tasks: identifying exclusively AI-generated reviews, detecting transition points in mixed reviews, and assessing the novelty of scientific work. We contribute to this domain by prompting open and closed source LLMs to generate data replicating the defined settings and conducting an in-depth analysis of automatic detectors for identifying machine-generated text. Furthermore, we innovatively repurpose an LLM to serve as an annotator for extracting novelty assessments from existing reviews. The annotated data obtained through this process will be leveraged as distant supervision to train an automated novelty assessment module in future work. Our findings demonstrate that LLMs exhibit moderate reliability in accomplishing this task, highlighting their potential for supporting the development of advanced review analysis tools. This study provides valuable insights and observations to guide future research endeavors in this domain, ultimately aiming to enhance the efficiency and effectiveness of peer review while upholding the principles of scientific inquiry and discourse.
Citation
O. Afzal, "Exploring the Integration of Large Language Models in Peer Review: Challenges, Opportunities, and Implications", M.S. Thesis, Natural Language Processing, MBZUAI, Abu Dhabi, UAE, 2024.