Methods for convex (l0,l1)-smooth optimization: clipping, acceleration, and adaptivity
Gorbunov, Eduard Tupitsa, Nazarii Choudhury, Sayantan Aliev, Alen Richtarik, Peter Horvath, Samuel Takac, Martin
Gorbunov, Eduard
Tupitsa, Nazarii
Choudhury, Sayantan
Aliev, Alen
Richtarik, Peter
Horvath, Samuel
Takac, Martin
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Due to the non-smoothness of optimization problems in Machine Learning, generalized smoothness assumptions have been gaining a lot of attention in recent years. One of the most popular assumptions of this type is (L0,L1)-smoothness (Zhang et al., 2020b). In this paper, we focus on the class of (strongly) convex (L0,L1)-smooth functions and derive new convergence guarantees for several existing methods. In particular, we derive improved convergence rates for Gradient Descent with (Smoothed) Gradient Clipping and for Gradient Descent with Polyak Stepsizes. In contrast to the existing results, our rates do not rely on the standard smoothness assumption and do not suffer from the exponential dependency on the initial distance to the solution. We also extend these results to the stochastic case under the over-parameterization assumption, propose a new accelerated method for convex (L0,L1)-smooth optimization, and derive new convergence rates for Adaptive Gradient Descent (Malitsky & Mishchenko, 2020). © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.
Citation
Y. Yan, C.-M. Feng, W. Zuo, S. Khan, L. Zhu, and Y. Liu, “On the Importance of Language-driven Representation Learning for Heterogeneous Federated Learning,” International Conference on Representation Learning, vol. 2025, pp. 63789–63812, May 2025
Source
13th International Conference on Learning Representations, ICLR 2025
Conference
13th International Conference on Learning Representations, ICLR 2025
Keywords
Subjects
Source
13th International Conference on Learning Representations, ICLR 2025
Publisher
International Conference on Learning Representations, ICLR