VENUS: Variance Reduction via Curvature-Based Gradient Transport
Bohara, Sushil
Bohara, Sushil
Citations
Altmetric:
Author
Supervisor
Department
Machine Learning
Embargo End Date
2027-05-01
Type
Thesis
Date
2026
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Stochastic optimization methods in machine learning, such as STORM and MARS, reduce gradient variance through a recursive momentum estimator that tracks the gradient difference between consecutive iterates. In STORM, evaluating this difference on the same data sample cancels much of the stochastic noise, though it is not completely eliminated. MARS introduces an additional hyperparameter to control the strength of this correction. However, both methods share the same underlying structure: the correction remains stochastic gradient difference, introducing a residual variance term that grows with correction strength and cannot be suppressed by tuning the learning rate or momentum parameters. This thesis identifies this stochastic gradient difference transport as the central structural limitation of existing recursive variance reduction methods and proposes VENUS, a new gradient estimator designed to eliminate this variance penalty while preserving the bias-correcting intent of the recursive update.
VENUS uses Hessian-vector product computed at the previous iterate using the previous mini-batch, making the correction fully measurable with respect to past information and
independent of the current stochastic noise. A theoretical variance analysis is conducted under standard per-sample smoothness and bounded noise assumptions, deriving tight
bounds for both MARS and VENUS and characterising the resulting bias-variance trade-off. Convergence guarantees are established for nonconvex objectives, achieving O(T^{-1/2}) in the worst case and an improved O(T^{-2/3}) rate under a refined bias assumption.
Empirical validation suggests that the theoretical variance advantage of VENUS translates to improved optimization performance in practice, particularly on tasks where the loss
landscape is complex. This work motivates a new direction in optimization method design where curvature information serves a dual purpose: preconditioning and variance-reduced gradient tracking.
Citation
Bohara, Sushil, "VENUS: Variance Reduction via Curvature-Based Gradient Transport," M.S. Thesis, Machine Learning, MBZUAI, 2026.
Source
Conference
Keywords
stochastic optimization, VENUS, variance reduction, curvature-based gradient transport, gradient tracking, Hessian-vector product