Follow my instruction and spill the beans: scalable data extraction from retrieval-augmented generation systems
Qi, Zhenting Zhang, Hanlin Xing, Eric P. Kakade, Sham Lakkaraju, Hima
Qi, Zhenting
Zhang, Hanlin
Xing, Eric P.
Kakade, Sham
Lakkaraju, Hima
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies. Extending our study to production RAG models, GPTs, we design an attack that can cause datastore leakage with a near-perfect success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves. Code is available at this repository. © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.
Citation
Z. Qi, H. Zhang, E. Xing, S. Kakade, and H. Lakkaraju, “Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems,” International Conference on Representation Learning, vol. 2025, pp. 48733–48755, May 2025
Source
13th International Conference on Learning Representations, ICLR 2025
Conference
13th International Conference on Learning Representations, ICLR 2025
Keywords
Subjects
Source
13th International Conference on Learning Representations, ICLR 2025
Publisher
International Conference on Learning Representations, ICLR