Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring
Mu, Honglin He, Han Zhou, Yuxin Feng, Yunlong Xu, Yang Qin, Libo Shi, Xiaoming Liu, Zeming Han, Xudong Shi, Qi
Mu, Honglin
He, Han
Zhou, Yuxin
Feng, Yunlong
Xu, Yang
Qin, Libo
Shi, Xiaoming
Liu, Zeming
Han, Xudong
Shi, Qi
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large language model (LLM) safety is a critical issue, with numerous studies employing red team testing to enhance model security. Among these, jailbreak methods explore potential vulnerabilities by crafting malicious prompts that induce model outputs contrary to safety alignments. Existing black-box jailbreak methods often rely on model feedback, repeatedly submitting queries with detectable malicious instructions during the attack search process. Although these approaches are effective, the attacks may be intercepted by content moderators during the search process. We propose an improved transfer attack method that guides malicious prompt construction by locally training a mirror model of the target black-box model through benign data distillation. This method offers enhanced stealth, as it does not involve submitting identifiable malicious instructions to the target model during the search phase. Our approach achieved a maximum attack success rate of 92%, or a balanced value of 80% with an average of 1.5 detectable jailbreak queries per sample against GPT-3.5 Turbo on a subset of AdvBench. These results underscore the need for more robust defense mechanisms.
Citation
H. Mu et al., “Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring,” vol. 1, pp. 1784–1799, Jun. 2025, doi: 10.18653/V1/2025.NAACL-LONG.88
Source
Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies: Long Papers, NAACL-HLT 2025
Conference
2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2025
Keywords
Subjects
Source
2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2025
Publisher
Association for Computational Linguistics