Causal Information Prioritization for Efficient Reinforcement Learning
Cao, Hongye Feng, Fan Yang, Tianpei Huo, Jing Gao, Yang
Cao, Hongye
Feng, Fan
Yang, Tianpei
Huo, Jing
Gao, Yang
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Reinforcement Learning (RL) methods often suffer from sample inefficiency, one of the underlying reasons is that blind exploration strategies may neglect causal relationships among states, actions, and rewards. Although recent causal approaches aim to address this problem, they lack grounded modeling of reward-guided causal understanding of states and actions for goal orientation, thus impairing learning efficiency. To tackle this issue, we propose a novel method named Causal Information Prioritization (CIP) that improves sample efficiency by leveraging factored MDPs to infer causal relationships between different dimensions of states and actions with respect to rewards, enabling the prioritization of causal information. Specifically, CIP identifies and leverages causal relationships between states and rewards to execute counterfactual data augmentation to prioritize high-impact state features under the causal understanding of the environments. Moreover, CIP integrates a causality-aware empowerment learning objective, which significantly enhances the agent's execution of reward-guided actions for more efficient exploration in complex environments. To fully assess the effectiveness of CIP, we conduct extensive experiments across 39 tasks in 5 diverse continuous control environments, encompassing both locomotion and manipulation skills learning with pixel-based and sparse reward settings. Experimental results demonstrate that CIP consistently outperforms existing RL methods across a wide range of scenarios. The project page is https://sites.google.com/view/rl-cip/. © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.
Citation
H. Cao, F. Feng, T. Yang, J. Huo, and Y. Gao, “Causal Information Prioritization for Efficient Reinforcement Learning,” International Conference on Representation Learning, vol. 2025, pp. 75061–75097, May 2025
Source
13th International Conference on Learning Representations, ICLR 2025
Conference
13th International Conference on Learning Representations, ICLR 2025
Keywords
Subjects
Source
13th International Conference on Learning Representations, ICLR 2025
Publisher
International Conference on Learning Representations, ICLR