# Adaptive Braking for Mitigating Gradient Delay

Source: [Adaptive Braking for Mitigating Gradient Delay](<https://www.sota2.com/research/paper/68fa571d1e209131ce9a2ace>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T14:33:38.409Z

Authors: Abhinav Venigalla, Atli Kosson, Vitaliy Chiley, Urs K\\"oster

Published: 2020-07-02T21:26:27.000Z

## Abstract

Neural network training is commonly accelerated by using multiple synchronized workers to compute gradient updates in parallel. Asynchronous methods remove synchronization overheads and improve hardware utilization at the cost of introducing gradient delay, which impedes optimization and can lead to lower final model performance. We introduce Adaptive Braking (AB), a modification for momentum-based optimizers that mitigates the effects of gradient delay. AB dynamically scales the gradient based on the alignment of the gradient and the velocity. This can dampen oscillations along high curvature directions of the loss surface, stabilizing and accelerating asynchronous training. We show that applying AB on top of SGD with momentum enables training ResNets on CIFAR-10 and ImageNet-1k with delays $D \\geq$ 32 update steps with minimal drop in final test accuracy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2007.01397>)

[PDF](<https://arxiv.org/pdf/2007.01397.pdf>)