# Train faster, generalize better: Stability of stochastic gradient descent

Source: [Train faster, generalize better: Stability of stochastic gradient descent](<https://www.sota2.com/research/paper/68fa528e1e209131ce97beed>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:55:37.975Z

Authors: Moritz Hardt, Benjamin Recht, Yoram Singer

Published: 2015-09-03T19:53:40.000Z

## Abstract

We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions.   Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Nonconvex Optimization on Nonconvex Optimization Theoretical](<https://www.sota2.com/research/sota/nonconvex-optimization-on-nonconvex-optimization-theoretical.md>): Nonconvex Optimization; Nonconvex Optimization Theoretical; Generalization Error: 1.

- [Generalization Bound Estimation on Synthetic Gaussian Distribution Dataset](<https://www.sota2.com/research/sota/generalization-bound-estimation-on-synthetic-gaussian-distribution-dataset.md>): Generalization Bound Estimation; Synthetic Gaussian Distribution Dataset; Generalization Bound Value: 4.04.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1509.01240>)

[PDF](<https://arxiv.org/pdf/1509.01240.pdf>)
