A Reinforcement Learning Approach in Multi-Phase Second-Price Auction Design
Ai, Rui Lyu, Boxiang Wang, Zhaoran Yang, Zhuoran Jordan, Michael
Ai, Rui
Lyu, Boxiang
Wang, Zhaoran
Yang, Zhuoran
Jordan, Michael
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
We study reserve price optimization in multi-phase second price auctions, where the seller’s prior actions affect the bidders’ later valuations through a Markov Decision Process (MDP). Compared to the bandit setting in existing works, the setting in ours involves three challenges. First, from the seller’s perspective, we need to efficiently explore the environment in the presence of potentially untruthful bidders who aim to manipulate the seller’s policy. Second, we want to minimize the seller’s revenue regret when the market noise distribution is unknown. Third, the seller’s per-step revenue is an unknown, nonlinear random variable, and cannot even be directly observed from the environment but realized values. We propose a mechanism addressing all three challenges. To address the first challenge, we use a combination of a new technique named “buffer periods” and inspirations from Reinforcement Learning (RL) with low switching cost to limit bidders’ surplus from untruthful bidding, thereby incentivizing approximately truthful bidding. The second one is tackled by a novel algorithm that removes the need for pure exploration when the market noise distribution is unknown. The third challenge is resolved by an extension of LSVI-UCB, where we use the auction’s underlying structure to control the uncertainty of the revenue function. The three techniques culminate in the Contextual-LSVI-UCB-Buffer (CLUB) algorithm which achieves Õ(H<sup>5</sup>/<sup>2 √</sup>K) revenue regret, where K is the number of episodes and H is the length of each episode, when the market noise is known and Õ(H<sup>3√</sup>K) revenue regret when the noise is unknown with no assumptions on bidders’ truthfulness.
Citation
R. Ai, B. Lyu, Z. Wang, Z. Yang, M. Jordan, "A Reinforcement Learning Approach in Multi-Phase Second-Price Auction Design," Journal of Machine Learning Research, vol. 27, 2026,
Source
Journal of Machine Learning Research
Conference
Keywords
Subjects
Source
Publisher
Microtome Publishing