Annotating the Annotators: Analysis, Insights and Modelling from an Annotation Campaign on Persuasion Techniques Detection
Bassi, Davide Dimitrov, Dimitar Iliyanov D'Auria, Bernardo Alam, Firoj Hasanain, Maram Moro, Christian Orru, Luisa Turchi, Gian Piero Nakov, Preslav Da San Martino, Giovanni
Bassi, Davide
Dimitrov, Dimitar Iliyanov
D'Auria, Bernardo
Alam, Firoj
Hasanain, Maram
Moro, Christian
Orru, Luisa
Turchi, Gian Piero
Nakov, Preslav
Da San Martino, Giovanni
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Persuasion (or propaganda) techniques detection is a relatively novel task in Natural Language Processing (NLP). While there have already been a number of annotation campaigns, they have been based on heuristic guidelines, which have never been thoroughly discussed. Here, we present the first systematic analysis of a complex annotation task -detecting 22 persuasion techniques in memes-, for which we provided continuous expert oversight. The presence of an expert allowed us to critically analyze specific aspects of the annotation process. Among our findings, we show that inter-annotator agreement alone inadequately assessed annotation correctness. We thus define and track different error types, revealing that expert feedback shows varying effectiveness across error categories. This pattern suggests that distinct mechanisms underlie different kinds of misannotations. Based on our findings, we advocate for an expert oversight in annotation tasks and periodic quality audits. As an attempt to reduce the costs for this, we introduce a probabilistic model for optimizing intervention scheduling.
Citation
D. Bassi et al., “Annotating the Annotators: Analysis, Insights and Modelling from an Annotation Campaign on Persuasion Techniques Detection,” 2025. [Online]. Available: https://aclanthology.org/2025.findings-acl.922/
Source
Findings of the Association for Computational Linguistics
Conference
Findings of the Association for Computational Linguistics: ACL 2025
Keywords
Subjects
Source
Findings of the Association for Computational Linguistics: ACL 2025
Publisher
Association for Computational Linguistics