OFedED: One-Shot Federated Learning through Model Ensemble and Dataset Distillation
Li, Xuhui
Li, Xuhui
Author
Supervisor
Department
Machine Learning
Embargo End Date
2026-05-30
Type
Thesis
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
One-shot federated learning (FL) has emerged as a promising paradigm in distributed machine learning, offering significant advantages in communication efficiency and scalability by reducing the need for multiple rounds of client-server interactions. However, the way most existing methods aggregate local models and generate data on the server, remains arguably inadequate to leverage the knowledge of client data, especially when it exhibits great heterogeneity across clients. To see this, for the first time, we theoretically show that under mild assumptions for local data distillations, the one-shot FL that aggregates distilled client data and optimizes the global model on the server, can achieve performance close to that under the centralized training. This theoretical insight bridges the gap between the empirical success observed in prior studies and the underlying mechanisms driving such success. Inspired by our theoretical findings, we propose a novel and more effective one-shot FL algorithm, termed OFedED (One-shot Federated Learning with Ensemble Distillation). OFedED integrates data distillation with a global ensemble mechanism, enabling it to capture differences in data distributions across clients while robustly leveraging local knowledge. Specifically, the distillation process ensures that the server can effectively synthesize representative data from heterogeneous clients, while the ensemble mechanism enhances the global model’s robustness by combining diverse local insights. Furthermore, OFedED incorporates privacy-preserving techniques to ensure that the communication of locally distilled data adheres to strict privacy guarantees. We conduct extensive experiments to validate the effectiveness of OFedED across multiple datasets, including MNIST, CIFAR-10, CIFAR-100 and SVHN, as well as various neural network architectures such as ResNet. Our results demonstrate that OFedED significantly outperforms state-of-the-art methods, achieving improvements of up to 9.07% on MNIST and 8.09% on CIFAR-10. Additionally, we verify the robustness of OFedED under different server-client architectures, highlighting its adaptability and scalability in real-world FL settings.
Citation
Xuhui Li, “OFedED: One-Shot Federated Learning through Model Ensemble and Dataset Distillation,” Master of Science thesis, Machine Learning, MBZUAI, 2025.
Source
Conference
Keywords
One-shot Federated Learning, Dataset Distillation, Ensemble Modeling