# Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems

Source: [Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems](<https://www.sota2.com/research/paper/68fa7a951e209131cea840ea>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-25T05:00:07.533Z

Authors: Jihyun Lim, Junhyuk Jo, Chanhyeok Ko, Young Min Go, Jimin Hwa, Sunwoo Lee

Published: 2025-08-12T01:03:09.000Z

## Abstract

Most parallel neural network training methods assume homogeneous computing resources. For example, synchronous data-parallel SGD suffers from significant synchronization overhead under heterogeneous workloads, often forcing practitioners to rely only on the fastest devices (e.g., GPUs). In this work, we study local SGD for efficient parallel training on heterogeneous systems. We show that intentionally introducing bias in data sampling and model aggregation can effectively harmonize slower CPUs with faster GPUs. Our extensive empirical results demonstrate that a carefully controlled bias significantly accelerates local SGD while achieving comparable or even higher accuracy than synchronous SGD under the same epoch budget. For instance, our method trains ResNet20 on CIFAR-10 with 2 CPUs and 8 GPUs up to 32x faster than synchronous SGD, with nearly identical accuracy. These results provide practical insights into how to flexibly utilize diverse compute resources for deep learning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 ResNet20 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-resnet20-test.md>): Image Classification; CIFAR-10 ResNet20 (test); Accuracy: 91.95.

- [Image Classification on Oxford Flowers 102 WRN16 (test)](<https://www.sota2.com/research/sota/image-classification-on-oxford-flowers-102-wrn16-test.md>): Image Classification; Oxford Flowers 102 WRN16 (test); Accuracy: 88.04.

- [Image Classification on CIFAR-100 WRN-28-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-wrn-28-10-test.md>): Image Classification; CIFAR-100 WRN-28-10 (test); Top-1 Accuracy: 80.58.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2508.08540>)

[PDF](<https://arxiv.org/pdf/2508.08540.pdf>)