Fraud-R1: A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements
Yang, Shu Zhu, Shenzhe Wu, Zeyu Wang, Keyu Yao, Junchi Wu, Junchao Hu, Lijie Li, Mengdi Wong, Derek F. Wang, Di
Yang, Shu
Zhu, Shenzhe
Wu, Zeyu
Wang, Keyu
Yao, Junchi
Wu, Junchao
Hu, Lijie
Li, Mengdi
Wong, Derek F.
Wang, Di
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
With the increasing integration of large language models (LLMs) into real-world applications such as finance, e-commerce, and recommendation systems, their susceptibility to misinformation and adversarial manipulation poses significant risks. Existing fraud detection benchmarks primarily focus on single-turn classification tasks, failing to capture the dynamic nature of real-world fraud attempts. To address this gap, we introduce Fraud-R1, a challenging bilingual benchmark designed to assess LLMs’ ability to resist fraud and phishing attacks across five key fraud categories: Fraudulent Services, Impersonation, Phishing Scams, Fake Job Postings, and Online Relationships, covering subclasses. Our dataset comprises manually curated fraud cases from social media, news, phishing scam records, and prior fraud datasets.
Citation
S. Yang et al., “Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements,” 2025. [Online]. Available: https://aclanthology.org/2025.findings-acl.226/
Source
Findings of the Association for Computational Linguistics: NAACL 2025
Conference
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Keywords
Subjects
Source
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Publisher
Association for Computational Linguistics