# A Quadratic Synchronization Rule for Distributed Deep Learning

Source: [A Quadratic Synchronization Rule for Distributed Deep Learning](<https://www.sota2.com/research/paper/68fa72e01e209131cea591a8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:36:44.860Z

Authors: Xinran Gu, Kaifeng Lyu, Sanjeev Arora, Jingzhao Zhang, Longbo Huang

Published: 2023-10-22T21:38:57.000Z

## Abstract

In distributed deep learning with data parallelism, synchronizing gradients at each training step can cause a huge communication overhead, especially when many nodes work together to train large models. Local gradient methods, such as Local SGD, address this issue by allowing workers to compute locally for $H$ steps without synchronizing with others, hence reducing communication frequency. While $H$ has been viewed as a hyperparameter to trade optimization efficiency for communication cost, recent research indicates that setting a proper $H$ value can lead to generalization improvement. Yet, selecting a proper $H$ is elusive. This work proposes a theory-grounded method for determining $H$, named the Quadratic Synchronization Rule (QSR), which recommends dynamically setting $H$ in proportion to $\\frac{1}{\\eta^2}$ as the learning rate $\\eta$ decays over time. Extensive ImageNet experiments on ResNet and ViT show that local gradient methods with QSR consistently improve the test accuracy over other synchronization strategies. Compared with the standard data parallel training, QSR enables Local AdamW on ViT-B to cut the training time on 16 or 64 GPUs down from 26.7 to 20.2 hours or from 8.6 to 5.5 hours and, at the same time, achieves $1.16\\%$ or $0.84\\%$ higher top-1 validation accuracy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (test) (Error Rate and Communication Cost)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-error-rate-and-communication-cost.md>): Image Classification; CIFAR-10 (test); Test Error Rate: 4.21.

- [Image Classification on CIFAR-100 (test) (Error Rate and Communication Overhead)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test-error-rate-and-communication-overhead.md>): Image Classification; CIFAR-100 (test); Test Error Rate: 17.77.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2310.14423>)

[PDF](<https://arxiv.org/pdf/2310.14423.pdf>)