Droid: A resource suite for ai-generated code detection
Orel, Daniil Paul, Indraneil Gurevych, Iryna Nakov, Preslav
Orel, Daniil
Paul, Indraneil
Gurevych, Iryna
Nakov, Preslav
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
We present DroidCollection, the most extensive open data suite for training and evaluating machine-generated code detectors, comprising over a million code samples, seven programming languages, outputs from 43 coding models, and three real-world coding domains. Alongside fully AI-generated examples, our collection includes human-AI co-authored code, as well as adversarial examples explicitly crafted to evade detection. Subsequently, we develop DroidDetect, a suite of encoder-only detectors trained using a multi-task objective over DroidCollection. Our experiments show that existing detectors’ performance fails to generalise to diverse coding domains and programming languages outside of their narrow training data. We further demonstrate that while most detectors are easily compromised by humanising the output distributions using superficial prompting and alignment approaches, this problem can be easily amended by training on a small number of adversarial examples. Finally, we demonstrate the effectiveness of metric learning and uncertainty-based resampling as way to enhance detector training on possibly noisy distributions.
Citation
D. Orel, I. Paul, I. Gurevych, and P. Nakov, “Droid: A Resource Suite for AI-Generated Code Detection,” Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 31251–31277, 2025, doi: 10.18653/V1/2025.EMNLP-MAIN.1593
Source
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Conference
2025 Conference on Empirical Methods in Natural Language Processing
Keywords
AI-Generated Code Detection, Multi-language Code Benchmark, Adversarial Code Examples, Encoder-only Detection Models, Multi-task Training Objective, Domain Generalisation in Code Detection, Metric Learning for Code Safety, Uncertainty-based Resampling
Subjects
Source
2025 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics