# Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space Source: [Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:37:22.464Z Authors: Akihito Taya, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto Published: 2021-04-01T09:17:20.000Z ## Abstract This paper proposes a fully decentralized federated learning (FL) scheme for Internet of Everything (IoE) devices that are connected via multi-hop networks. Because FL algorithms hardly converge the parameters of machine learning (ML) models, this paper focuses on the convergence of ML models in function spaces. Considering that the representative loss functions of ML tasks e.g, mean squared error (MSE) and Kullback-Leibler (KL) divergence, are convex functionals, algorithms that directly update functions in function spaces could converge to the optimal solution. The key concept of this paper is to tailor a consensus-based optimization algorithm to work in the function space and achieve the global optimum in a distributed manner. This paper first analyzes the convergence of the proposed algorithm in a function space, which is referred to as a meta-algorithm, and shows that the spectral graph theory can be applied to the function space in a manner similar to that of numerical vectors. Then, consensus-based multi-hop federated distillation (CMFD) is developed for a neural network (NN) to implement the meta-algorithm. CMFD leverages knowledge distillation to realize function aggregation among adjacent devices without parameter averaging. An advantage of CMFD is that it works even with different NN models among the distributed learners. Although CMFD does not perfectly reflect the behavior of the meta-algorithm, the discussion of the meta-algorithm's convergence property promotes an intuitive understanding of CMFD, and simulation evaluations show that NN models converge using CMFD for several tasks. The simulation results also show that CMFD achieves higher accuracy than parameter aggregation for weakly connected networks, and CMFD is more stable than parameter aggregation methods. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on F-MNIST 500x2](): Image Classification; F-MNIST 500x2; Final Accuracy: 78.4. - [Image Classification on F-MNIST 1000x1](): Image Classification; F-MNIST 1000x1; Accuracy: 77.2. - [Image Classification on CIFAR-10 2500x2](): Image Classification; CIFAR-10 2500x2; Final Accuracy: 45.6. - [Image Classification on CIFAR-10 (5000x1)](): Image Classification; CIFAR-10 (5000x1); Final Accuracy: 51.7. - [Image Classification on F-MNIST Dirichlet](): Image Classification; F-MNIST Dirichlet; Final Accuracy (%): 84. - [Image Classification on CIFAR-10 Dirichlet](): Image Classification; CIFAR-10 Dirichlet; Final Accuracy: 63.2. ## Original sources and implementation [Original paper]() [PDF]()