# FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning Source: [FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning]() Collection index: [llms.txt]() Data updated: 2025-10-26T01:59:40.801Z Authors: Huy Q. Le, Minh N. H. Nguyen, Chu Myaet Thwal, Yu Qiao, Chaoning Zhang, Choong Seon Hong Published: 2023-07-25T02:55:33.000Z ## Abstract Federated learning (FL) enables a decentralized machine learning paradigm for multiple clients to collaboratively train a generalized global model without sharing their private data. Most existing works simply propose typical FL systems for single-modal data, thus limiting its potential on exploiting valuable multimodal data for future personalized applications. Furthermore, the majority of FL approaches still rely on the labeled data at the client side, which is limited in real-world applications due to the inability of self-annotation from users. In light of these limitations, we propose a novel multimodal FL framework that employs a semi-supervised learning approach to leverage the representations from different modalities. Bringing this concept into a system, we develop a distillation-based multimodal embedding knowledge transfer mechanism, namely FedMEKT, which allows the server and clients to exchange the joint knowledge of their learning models extracted from a small multimodal proxy dataset. Our FedMEKT iteratively updates the generalized global encoders with the joint embedding knowledge from the participating clients. Thereby, to address the modality discrepancy and labeled data constraint in existing FL systems, our proposed FedMEKT comprises local multimodal autoencoder learning, generalized multimodal autoencoder construction, and generalized classifier learning. Through extensive experiments on three multimodal human activity recognition datasets, we demonstrate that FedMEKT achieves superior global encoder performance on linear evaluation and guarantees user privacy for personal data and model parameters while demanding less communication cost than other baselines. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image-to-Text Retrieval on IAPR TC-12 (test)](): Image-to-Text Retrieval; IAPR TC-12 (test); R@1: 28.18. - [Text-to-Image Retrieval on IAPR TC-12 (test)](): Text-to-Image Retrieval; IAPR TC-12 (test); R@1: 29.53. - [Text-to-Image Retrieval on MS-COCO (test)](): Text-to-Image Retrieval; MS-COCO (test); R@1: 35.92. - [Image-to-text Retrieval on MS-COCO (test)](): Image-to-Text Retrieval; MS-COCO (test); R@1: 34.86. - [Text-to-Video Retrieval on MSR-VTT (test)](): Text-to-Video Retrieval; MSR-VTT (test); R@1: 25.38. - [Image-to-Text Retrieval on Flickr30k (test)](): Image-to-Text Retrieval; Flickr30k (test); R@1: 37.46. - [Text-to-Image Retrieval on Flickr30k (test)](): Text-to-Image Retrieval; Flickr30k (test); Recall@1: 39.15. ## Original sources and implementation [Original paper]() [PDF]()