# FedUMM: A General Framework for Federated Learning with Unified Multimodal Models

Source: [FedUMM: A General Framework for Federated Learning with Unified Multimodal Models](<https://www.sota2.com/research/paper/69730092b2be0ed85a78ca68>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-24T05:00:05.231Z

Authors: Zhaolong Su, Leheng Zhao, Xiaoying Wu, Ziyue Xu, Jindong Wang

Published: 2026-01-21T19:02:52.000Z

## Abstract

Unified multimodal models (UMMs) are emerging as strong foundation models that can do both generation and understanding tasks in a single architecture. However, they are typically trained in centralized settings where all training and downstream datasets are gathered in a central server, limiting the deployment in privacy-sensitive and geographically distributed scenarios. In this paper, we present FedUMM, a general federated learning framework for UMMs under non-IID multimodal data with low communication cost. Built on NVIDIA FLARE, FedUMM instantiates federation for a BLIP3o backbone via parameter-efficient fine-tuning: clients train lightweight LoRA adapters while freezing the foundation models, and the server aggregates only adapter updates. We evaluate on VQA v2 and the GenEval compositional generation benchmarks under Dirichlet-controlled heterogeneity with up to 16 clients. Results show slight degradation as client count and heterogeneity increase, while remaining competitive with centralized training. We further analyze computation--communication trade-offs and demonstrate that adapter-only federation reduces per-round communication by over an order of magnitude compared to full fine-tuning, enabling practical federated UMM training. This work provides empirical experience for future research on privacy-preserving federated unified multimodal models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Question Answering on VQA v2 (Accuracy & Efficiency)](<https://www.sota2.com/research/sota/visual-question-answering-on-vqa-v2-accuracy-and-efficiency.md>): Visual Question Answering; VQA v2; Accuracy: 80.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.15390>)

[PDF](<https://arxiv.org/pdf/2601.15390.pdf>)
