# Demon: Improved Neural Network Training with Momentum Decay

Source: [Demon: Improved Neural Network Training with Momentum Decay](<https://www.sota2.com/research/paper/68fa583c1e209131ce9ac9d3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T05:18:49.253Z

Authors: John Chen, Cameron Wolfe, Zhao Li, Anastasios Kyrillidis

Published: 2019-10-11T03:12:21.000Z

## Abstract

Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\\textsc{Demon}) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate decay schedules. Across 28 relevant combinations of models, epochs, datasets, and optimizers, \\textsc{Demon} achieves the highest number of Top-1 and Top-3 finishes at 39\\% and 85\\% respectively, almost doubling the second-placed learning rate cosine schedule at 17\\% and 60\\%, respectively. \\textsc{Demon} also outperforms other widely used schedulers including, but not limited to, the learning rate step schedule, linear schedule, OneCycle schedule, and exponential schedule. Compared with the widely used learning rate step schedule, \\textsc{Demon} is observed to be less sensitive to parameter tuning, which is critical to training neural networks in practice. Results are demonstrated across a variety of settings and architectures, including image classification, generative models, and language models. \\textsc{Demon} is easy to implement, requires no additional tuning, and incurs almost no extra computational overhead compared to the vanilla counterparts. Code is readily available.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1910.04952>)

[PDF](<https://arxiv.org/pdf/1910.04952.pdf>)
