FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning
Nurbek, Tastan Samuel, Horvath Martin, Takac Karthik, Nandakumar
Nurbek, Tastan
Samuel, Horvath
Martin, Takac
Karthik, Nandakumar
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is extreme data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This personalized warmup allows the participants to focus initially on learning specific subnetworks tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (FedPeWS) approach improves accuracy and convergence speed over standard federated optimization methods. The code can be found at https://github.com/tnurbek/fedpews.
Citation
N. Tastan, S. Horváth, M. Takáč, and K. Nandakumar, “FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning,” in Proc. 2nd Conf. Parsimony and Learning (CPAL), Stanford, CA, USA, Mar. 24–27, 2025, Proc. Mach. Learn. Res., vol. 280, pp. 462–483. ML Research Press, 2025
Source
Proceedings of Machine Learning Research
Conference
2nd Conference on Parsimony and Learning, CPAL 2025
Keywords
Data Accuracy, Data Aggregation, Learning Systems, Machine Learning, Optimization, Convergence Speed, Data Heterogeneity, Full Model, Learn+, Optimisations, Optimization Method, Subnetworks, Federated Learning
Subjects
Source
2nd Conference on Parsimony and Learning, CPAL 2025
Publisher
ML Research Press