# Clip21: Error Feedback for Gradient Clipping

Source: [Clip21: Error Feedback for Gradient Clipping](<https://www.sota2.com/research/paper/68fa5a711e209131ce9bfed8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T16:30:44.528Z

Authors: Sarit Khirirat, Eduard Gorbunov, Samuel Horv\\'ath, Rustem Islamov, Fakhri Karray, Peter Richt\\'arik

Published: 2023-05-30T10:41:42.000Z

## Abstract

Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i.e., clipping applied to the gradients computed from local information at the nodes. While gradient clipping is an essential tool for injecting formal DP guarantees into gradient-based methods \[1\], it also induces bias which causes serious convergence issues specific to the distributed setting. Inspired by recent progress in the error-feedback literature which is focused on taming the bias/error introduced by communication compression operators such as Top-$k$ \[2\], and mathematical similarities between the clipping operator and contractive compression operators, we design Clip21 -- the first provably effective and practically useful error feedback mechanism for distributed methods with gradient clipping. We prove that our method converges at the same $\\mathcal{O}\\left(\\frac{1}{K}\\right)$ rate as distributed gradient descent in the smooth nonconvex regime, which improves the previous best $\\mathcal{O}\\left(\\frac{1}{\\sqrt{K}}\\right)$ rate which was obtained under significantly stronger assumptions. Our method converges significantly faster in practice than competing methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on MNIST (test) (Epsilon-based Robustness Evaluation)](<https://www.sota2.com/research/sota/image-classification-on-mnist-test-epsilon-based-robustness-evaluation.md>): Image Classification; MNIST (test); Accuracy (epsilon=3): 49.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.18929>)

[PDF](<https://arxiv.org/pdf/2305.18929.pdf>)