# U-Clip: On-Average Unbiased Stochastic Gradient Clipping

Source: [U-Clip: On-Average Unbiased Stochastic Gradient Clipping](<https://www.sota2.com/research/paper/68fa5a111e209131ce9bcb6a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T16:17:39.029Z

Authors: Bryn Elesedy, Marcus Hutter

Published: 2023-02-06T18:01:38.000Z

## Abstract

U-Clip is a simple amendment to gradient clipping that can be applied to any iterative gradient optimization algorithm. Like regular clipping, U-Clip involves using gradients that are clipped to a prescribed size (e.g. with component wise or norm based clipping) but instead of discarding the clipped portion of the gradient, U-Clip maintains a buffer of these values that is added to the gradients on the next iteration (before clipping). We show that the cumulative bias of the U-Clip updates is bounded by a constant. This implies that the clipped updates are unbiased on average. Convergence follows via a lemma that guarantees convergence with updates $u\_i$ as long as $\\sum\_{i=1}^t (u\_i - g\_i) = o(t)$ where $g\_i$ are the gradients. Extensive experimental exploration is performed on CIFAR10 with further validation given on ImageNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2302.02971>)

[PDF](<https://arxiv.org/pdf/2302.02971.pdf>)