Learning from Delay Distributions: A New Representation for Delay-Aware Reinforcement Learning
Yu, Zhuoru Fu, Chenchen Zhong, Hengkai Wang, Wanyuan Wu, Weiwei Xue, Chun Jason
Yu, Zhuoru
Fu, Chenchen
Zhong, Hengkai
Wang, Wanyuan
Wu, Weiwei
Xue, Chun Jason
Supervisor
Department
Computer Science
Embargo End Date
Type
Conference proceeding
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Delay remains a significant challenge for applying deep reinforcement learning (DRL) in real-world scenarios. Existing delay-aware DRL methods primarily rely on state augmentation to restore the Markov property in delayed environments, yet often assume the prior knowledge of the exact delay values and suffer from performance degradation in random delay environments. However, we observe that the random delays in real-world often follow specific statistical patterns. Based on this observation, we propose a novel method that leverages distributions to represent value functions, enabling a more accurate modeling of delay uncertainty beyond traditional expectation-based methods. Building upon delay distribution properties, we introduce a stochastic delay representation mechanism to reconstruct precise returns in delayed environments and prove its convergence to the optimal policy. Finally, we apply these techniques to design the delay-aware distributional actor-critic (D2 AC) DRL framework. Experimental results show that D2 AC significantly outperforms state-of-the-art delay-aware DRL methods across various random delay distributions in MuJoCo continuous control tasks. Open source code and appendix are available at: https://github.com/COOLAS-CS/D2AC.
Citation
Z. Yu, C. Fu, H. Zhong, W. Wang, W. Wu, C.J. Xue, "Learning from Delay Distributions: A New Representation for Delay-Aware Reinforcement Learning," 2026, pp. 459-467.
Source
Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems
Conference
AAMAS 2026: Autonomous Agents and Multiagent Systems
Keywords
46 Information and Computing Sciences, 4602 Artificial Intelligence, 4611 Machine Learning
Subjects
Source
AAMAS 2026: Autonomous Agents and Multiagent Systems
Publisher
International Foundation for Autonomous Agents and Multiagent Systems