# FedMultimodal: A Benchmark For Multimodal Federated Learning Source: [FedMultimodal: A Benchmark For Multimodal Federated Learning]() Collection index: [llms.txt]() Data updated: 2025-10-26T01:26:24.585Z Authors: Tiantian Feng, Digbalay Bose, Tuo Zhang, Rajat Hebbar, Anil Ramakrishna, Rahul Gupta, Mi Zhang, Salman Avestimehr, Shrikanth Narayanan Published: 2023-06-15T20:31:26.000Z ## Abstract Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Emotion Recognition on MELD (Relative Performance Degradation)](): Emotion Recognition; MELD; Relative Performance Degradation: 4. - [Relative Performance Degradation on UCI-HAR](): Human Activity Recognition; UCI-HAR; Relative Degradation: 4.6. - [Electrocardiography Classification on PTB-XL](): Electrocardiography Classification; PTB-XL; Relative Performance Degradation: 14. - [Relative Performance Degradation on KU-HAR](): Human Activity Recognition; KU-HAR; Relative Performance Degradation: 13. - [Multimodal Emotion Recognition on MELD Natural Distribution](): Multimodal Emotion Recognition; MELD Natural Distribution; Accuracy: 49.39. - [Electrocardiography Classification on PTB-XL Natural Distribution](): Electrocardiography Classification; PTB-XL Natural Distribution; Accuracy: 16.4. - [Multimodal Emotion Recognition on MELD IID Setting](): Multimodal Emotion Recognition; MELD IID Setting; Accuracy: 49.39. - [Multimodal Federated Learning on ActionSense 100 communication rounds](): Multimodal Federated Learning; ActionSense 100 communication rounds; Training Time (mins): 29.44. - [Action Recognition on ActionSense IID Setting](): Action Recognition; ActionSense IID Setting; Accuracy: 13.45. - [Action Recognition on ActionSense Natural Distribution](): Action Recognition; ActionSense Natural Distribution; Accuracy: 24.81. - [Electrocardiography Classification on PTB-XL IID Setting](): Electrocardiography Classification; PTB-XL IID Setting; Accuracy: 4.07. - [Human Activity Recognition on UCI-HAR IID Setting](): Human Activity Recognition; UCI-HAR IID Setting; Accuracy: 17.52. - [Human Activity Recognition on UCI-HAR Natural Distribution](): Human Activity Recognition; UCI-HAR Natural Distribution; Accuracy (%): 17.7. - [Deepfake Detection on DFC IID Setting 23](): Deepfake Detection; DFC IID Setting 23; Accuracy: 62.4. ## Original sources and implementation [Original paper]() [PDF]()