FLECS: a federated learning second-order framework via compression and sketching
Agafonov, Artem D. Kamzolov, Dmitry I. Tappenden, Rachael E.H. Gasnikov, Alexander V. Takac, Martin
Agafonov, Artem D.
Kamzolov, Dmitry I.
Tappenden, Rachael E.H.
Gasnikov, Alexander V.
Takac, Martin
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Inspired by the recent work FedNL [Safaryan et al. FedNL: making newton-type methods applicable to federated learning], we propose a new communication efficient second-order framework for Federated learning, namely FLECS. The proposed method reduces the high-memory requirements of FedNL by the usage of an L-SR1 type update for the Hessian approximation which is stored on the central server. A low dimensional ‘sketch’ of the Hessian is all that is needed by each device to generate an update, so that memory costs as well as number of Hessian-vector products for the agent are low. Biased and unbiased compressions are utilized to make communication costs also low. Convergence guarantees for FLECS are provided in both the strongly convex, and non-convex cases, and local linear convergence is also established under strong convexity. Numerical experiments confirm the practical benefits of this new FLECS algorithm.
Citation
A. Agafonov, D. Kamzolov, R. Tappenden, A. Gasnikov, and M. Takáč, “FLECS: a federated learning second-order framework via compression and sketching,” Optim Methods Softw, Aug. 2025, doi: 10.1080/10556788.2025.2456118
Source
Optimization Methods and Software
Conference
Keywords
Compression, Convex Optimization, Distributed Optimization, Federated Learning, Non-convex Optimization, Quasi-newton Methods, Second-order Methods, Sketching, Computational Methods, Newton-raphson Method, Compression, Convex Optimisation, Distributed Optimization, Memory Requirements, Newton-type Methods, Nonconvex Optimization, Quasi-newton Methods, Second Orders, Second-order Methods, Sketchings, Convex Optimization
Subjects
Source
Publisher
Taylor & Francis