# Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study

Source: [Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study](<https://www.sota2.com/research/paper/68fa62fe1e209131ce9fc189>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:32.892Z

Authors: Suyog Gupta, Wei Zhang, Fei Wang

Published: 2015-09-14T17:14:52.000Z

## Abstract

This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learning rate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classification benchmarks: CIFAR10 and ImageNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1509.04210>)

[PDF](<https://arxiv.org/pdf/1509.04210.pdf>)