# Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models Source: [Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models]() Collection index: [llms.txt]() Data updated: 2026-06-06T05:00:11.965Z Authors: Sunny Gupta, Shambhavi Shanker, Amit Sethi Published: 2026-06-04T13:28:48.000Z ## Abstract Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning. However, existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, where independently averaging low rank factors fails to approximate the true combined update, and (2) client side initialization lag, as clients repeatedly reinitialize LoRA parameters across communication rounds, slowing convergence. We propose HyperLoRA, a unified framework that addresses both issues through amortized federated adaptation through hypernetwork-driven LoRA generation and product space aggregation. Instead of iterative per-client optimization, HyperLoRA employs a learned generator that maps client distribution signatures to LoRA initializations, effectively amortizing per client adaptation. On the server side, we introduce a learned aggregation module that directly synthesizes updates in the low-rank product space, eliminating the inconsistencies of factor-wise averaging. A lightweight residual correction module further improves stability under heterogenous (non-IID) client distributions.By replacing iterative optimization and heuristic averaging with learned operators, HyperLoRA jointly enables efficient personalization, unbiased aggregation, and faster convergence. Experiments on federated vision and vision-language benchmarks show that HyperLoRA achieves improved convergence speed, greater robustness to distribution shift, and stronger personalization performance compared to prior federated LoRA methods. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on DomainNet Feature non-IID](): Image Classification; DomainNet Feature non-IID; Top-1 Accuracy: 78.01. - [Image Classification on NICO++ Feature non-IID](): Image Classification; NICO++ Feature non-IID; Top-1 Accuracy: 92.91. - [Image Classification on DomainNet (Feature+label non-IID Average)](): Image Classification; DomainNet Feature+label non-IID (average across domains); Top-1 Acc: 76.11. - [Image Classification on NICO++ Feature+label non-IID (average across domains)](): Image Classification; NICO++ Feature+label non-IID (average across domains); Top-1 Accuracy: 91.19. ## Original sources and implementation [Original paper]() [PDF]()