# Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients Source: [Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients]() Collection index: [llms.txt]() Data updated: 2026-01-02T10:11:45.444Z Authors: Jabin Koo, Minwoo Jang, Jungseul Ok Published: 2024-10-30T08:48:21.000Z ## Abstract Federated fine-tuning for Large Language Models (LLMs) faces significant challenges due to the heavy communication overhead of transmitting large model updates. Although Low Rank Adaptation (LoRA) has been proposed as a solution, yet its application in federated learning is complicated by discordance in aggregation. Existing methods addressing this discordance often suffer from performance degradation at low ranks in heterogeneous data settings. In response, we introduce LoRA-A$^2$ (Low Rank Adaptation with Alternating freeze and Adaptive rank selection), which demonstrates robustness in challenging settings with low ranks and high data heterogeneity. Our experimental findings reveal that LoRA-A$^2$ maintains performance even under extreme heterogeneity and low rank conditions, achieving up to a significant reduction in uploaded parameters compared to full fine-tuning without compromising performance. This adaptive mechanism increases robustness and communication efficiency in federated fine-tuning, enabling the practical deployment of LLMs in resource-constrained environments. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Text Classification on BANKING77 Dir(0.01) (test)](): Text Classification; BANKING77 Dir(0.01) (test); Accuracy: 85.98. - [Text Classification on BANKING77 Dir(0.5) (test)](): Text Classification; BANKING77 Dir(0.5) (test); Accuracy: 93.24. - [Text Classification on BANKING77 Dir(0.1) (test)](): Text Classification; BANKING77 Dir(0.1) (test); Accuracy: 92.02. - [Text Classification on 20 Newsgroups Dir(0.01) (test)](): Text Classification; 20 Newsgroups Dir(0.01) (test); Accuracy: 0.5484. - [Text Classification on 20 Newsgroups Dir(0.5) (test)](): Text Classification; 20 Newsgroups Dir(0.5) (test); Accuracy: 70.31. - [Text Classification on 20 Newsgroups Dir(0.1) (test)](): Text Classification; 20 Newsgroups Dir(0.1) (test); Accuracy: 67.12. ## Original sources and implementation [Original paper]() [PDF]()