Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles
Nwadike, Munachiso S Iklassov, Zangir Aremu, Toluwani Hiraoka, Tatsuya Heinzerling, Benjamin Bojkovic, Velibor Alquabeh, Hilal Takac, Martin Inui, Kentaro
Nwadike, Munachiso S
Iklassov, Zangir
Aremu, Toluwani
Hiraoka, Tatsuya
Heinzerling, Benjamin
Bojkovic, Velibor
Alquabeh, Hilal
Takac, Martin
Inui, Kentaro
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
We introduce the concept of the self-referencing causal cycle (abbreviated ReCall )—a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomenon known as the reversal curse. When an LLM is prompted with sequential data, it often fails to recall preceding context. For example, when we ask an LLM to recall the line preceding “O say does that star-spangled banner yet wave” in the U.S. National Anthem, it often fails to correctly return “Gave proof through the night that our flag was still there”—this is due to the reversal curse. It occurs because language models such as ChatGPT and Llama generate text based on preceding tokens, requiring facts to be learned and reproduced in a consistent token order. While the reversal curse is often viewed as a limitation, we offer evidence of an alternative view: it is not always an obstacle in practice. We find that ReCall is driven by what we designate as cycle tokens—sequences that connect different parts of the training data, enabling recall of preceding tokens from succeeding ones. Through rigorous probabilistic formalization and controlled experiments, we demonstrate how the cycles they induce influence a model’s ability to reproduce information. To facilitate reproducibility, we provide our code and experimental details at https://anonymous.4open.science/r/remember-B0B8/.
Citation
J. Wu et al., “Medical Graph RAG: Evidence-based Medical Large Language Model via Graph Retrieval-Augmented Generation,” 2025. [Online]. Available: https://aclanthology.org/2025.acl-long.1381/
Source
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics
Conference
63rd Annual Meeting of the Association for Computational Linguistics, 2025
Keywords
Subjects
Source
63rd Annual Meeting of the Association for Computational Linguistics, 2025
Publisher
Association for Computational Linguistics