# Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning

Source: [Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning](<https://www.sota2.com/research/paper/68fa38671e209131ce8ba4de>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T03:53:38.362Z

Authors: Yao Zhang, Andrew M. Saxe, Madhu S. Advani, Alpha A. Lee

Published: 2018-03-05T21:12:04.000Z

## Abstract

Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy-entropy competition in statistical physics, wide but shallow minima can be optimal if the system is undersampled, as is typical in many applications. Moreover, we show that the stochasticity in the algorithm has a non-trivial correlation structure which systematically biases it towards wide minima. We illustrate our argument with two prototypical models: image classification using deep learning, and a linear neural network where we can analytically reveal the relationship between entropy and out-of-sample error.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1803.01927>)

[PDF](<https://arxiv.org/pdf/1803.01927.pdf>)
