GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge
Dugan, Liam Zhu, Andrew Alam, Firoj Nakov, Preslav Apidianaki, Marianna Callison-Burch, Chris
Dugan, Liam
Zhu, Andrew
Alam, Firoj
Nakov, Preslav
Apidianaki, Marianna
Callison-Burch, Chris
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recently there have been many shared tasks targeting the detection of generated text from Large Language Models (LLMs). However, these shared tasks tend to focus either on cases where text is limited to one particular domain or cases where text can be from many domains, some of which may not be seen during test time. In this shared task, using the newly released RAID benchmark, we aim to answer whether or not models can detect generated text from a large, yet fixed, number of domains and LLMs, all of which are seen during training. Over the course of three months, our task was attempted by 9 teams with 23 detector submissions. We find that multiple participants were able to obtain accuracies of over 99% on machine-generated text from RAID while maintaining a 5% False Positive Rate-suggesting that detectors are able to robustly detect text from many domains and models simultaneously. We discuss potential interpretations of this result and provide directions for future research.
Citation
L. Dugan, A. Zhu, F. Alam, P. Nakov, M. Apidianaki, and C. Callison-Burch, “GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge,” Proceedings - International Conference on Computational Linguistics, COLING, pp. 377–388, Jan. 2025.
Source
Proceedings - International Conference on Computational Linguistics, COLING
Conference
Keywords
Cross-domain machine-generated text detection, Large Language Models (LLMs), RAID benchmark, Detection accuracy, False positive rate?
Subjects
Source
Publisher
Association for Computational Linguistics