Enhancing Policy Gradient with the Polyak Step-Size Adaption
Li, Yunxiang
Li, Yunxiang
Author
Supervisor
Department
Machine Learning
Embargo End Date
2024-01-01
Type
Thesis
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
IPolicy gradient stands as a cornerstone within the realm of reinforcement learning (RL), revered for its widespread adoption and foundational significance. While celebrated for its convergence guarantees and stability relative to other RL algorithms, its pragmatic utility often encounters roadblocks stemming from hyper-parameter sensitivity, notably the stepsize. In this manuscript, we unveil a groundbreaking advancement in RL methodology by introducing the integration of the Polyak step-size, a mechanism designed to autonomously adjust the step-size without necessitating prior knowledge. Our endeavor to adapt this method to RL settings involves addressing many challenges, chief among them being the presence of unknown f? in the Polyak step-size formulation. Moreover, we present empirical evaluation of the Polyak step-size within RL frameworks through designed experiments. The outcomes of our empirical analyses serve to illuminate the better performance of the Polyak step-size, showcasing its propensity for facilitating expedited convergence and the realization of more stable policies in diverse RL environments.
Citation
Y. Li, "Enhancing Policy Gradient with the Polyak Step-Size Adaption", MS. Thesis, Machine Learning, MBZUAI, Abu Dhabi, UAE, 2024