HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
Lin, Zheng Zhang, Yuxin Chen, Zhe Fang, Zihan Chen, Xianhao Vepakomma, Praneeth Ni, Wei Luo, Jun Gao, Yue
Lin, Zheng
Zhang, Yuxin
Chen, Zhe
Fang, Zihan
Chen, Xianhao
Vepakomma, Praneeth
Ni, Wei
Luo, Jun
Gao, Yue
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recently, large language models (LLMs) have achieved remarkable breakthroughs, revolutionizing the natural language processing domain and beyond. Due to immense parameter sizes, fine-tuning these models with private data for diverse downstream tasks has become mainstream. Though federated learning (FL) offers a promising solution for fine-tuning LLMs without sharing raw data, substantial computing costs hinder its democratization. Moreover, in real-world scenarios, private client devices often possess heterogeneous computing resources, further complicating LLM fine-tuning. To combat these challenges, we propose HSplitLoRA, a heterogeneous parameter-efficient fine-tuning (PEFT) framework built on split learning (SL) and low-rank adaptation (LoRA) fine-tuning, for efficiently fine-tuning LLMs on heterogeneous client devices. HSplitLoRA first identifies important weights based on their contributions to LLM training. It then dynamically configures the decomposition ranks of LoRA adapters for selected weights and determines the model split point according to varying computing budgets of client devices. Finally, a noise-free adapter aggregation mechanism is devised to support heterogeneous adapter aggregation without introducing noise. Extensive experiments demonstrate that HSplitLoRA outperforms state-of-the-art benchmarks in training accuracy and convergence speed.
Citation
Z. Lin, Y. Zhang, Z. Chen, Z. Fang, X. Chen, P. Vepakomma , et al., "HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models," IEEE Transactions on Mobile Computing, vol. PP, no. 99, pp. 1-17, https://doi.org/10.1109/tmc.2026.3680521.
Source
IEEE Transactions on Mobile Computing
Conference
Keywords
46 Information and Computing Sciences, 4605 Data Management and Data Science, 4611 Machine Learning
Subjects
Source
Publisher
IEEE