Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization
Demidovich, Yury Ostroukhov, Petr Malinovsky, Grigory Horvath, Samuel Takac, Martin Richtarik, Peter Gorbunov, Eduard
Demidovich, Yury
Ostroukhov, Petr
Malinovsky, Grigory
Horvath, Samuel
Takac, Martin
Richtarik, Peter
Gorbunov, Eduard
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Non-convex Machine Learning problems typically do not adhere to the standard smoothness assumption. Based on empirical findings, Zhang et al. (2020b) proposed a more realistic generalized (L0, L1)-smoothness assumption, though it remains largely unexplored. Many existing algorithms designed for standard smooth problems need to be revised. However, in the context of Federated Learning, only a few works address this problem but rely on additional limiting assumptions. In this paper, we address this gap in the literature: we propose and analyze new methods with local steps, partial participation of clients, and Random Reshuffling without extra restrictive assumptions beyond generalized smoothness. The proposed methods are based on the proper interplay between clients' and server's stepsizes and gradient clipping. Furthermore, we perform the first analysis of these methods under the Polyak-Łojasiewicz condition. Our theory is consistent with the known results for standard smooth problems, and our experimental results support the theoretical insights. © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.
Citation
Y. Demidovich et al., “Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization,” International Conference on Representation Learning, vol. 2025, pp. 80916–80983, May 2025
Source
13th International Conference on Learning Representations, ICLR 2025
Conference
13th International Conference on Learning Representations, ICLR 2025
Keywords
Subjects
Source
13th International Conference on Learning Representations, ICLR 2025
Publisher
International Conference on Learning Representations, ICLR