# AdaGossip: Adaptive Consensus Step-size for Decentralized Deep Learning with Communication Compression

Source: [AdaGossip: Adaptive Consensus Step-size for Decentralized Deep Learning with Communication Compression](<https://www.sota2.com/research/paper/68fa72cf1e209131cea58a43>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:34:49.549Z

Authors: Sai Aparna Aketi, Abolfazl Hashemi, Kaushik Roy

Published: 2024-04-09T00:43:45.000Z

## Abstract

Decentralized learning is crucial in supporting on-device learning over large distributed datasets, eliminating the need for a central server. However, the communication overhead remains a major bottleneck for the practical realization of such decentralized setups. To tackle this issue, several algorithms for decentralized training with compressed communication have been proposed in the literature. Most of these algorithms introduce an additional hyper-parameter referred to as consensus step-size which is tuned based on the compression ratio at the beginning of the training. In this work, we propose AdaGossip, a novel technique that adaptively adjusts the consensus step-size based on the compressed model differences between neighboring agents. We demonstrate the effectiveness of the proposed method through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, Imagenette, and ImageNet), model architectures, and network topologies. Our experiments show that the proposed method achieves superior performance ($0-2\\%$ improvement in test accuracy) compared to the current state-of-the-art method for decentralized learning with communication compression.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.05919>)

[PDF](<https://arxiv.org/pdf/2404.05919.pdf>)
