# The AdEMAMix Optimizer: Better, Faster, Older

Source: [The AdEMAMix Optimizer: Better, Faster, Older](<https://www.sota2.com/research/paper/68fa74f71e209131cea64ede>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:22:19.696Z

Authors: Matteo Pagliardini, Pierre Ablin, David Grangier

Published: 2024-09-05T00:13:16.000Z

## Abstract

Momentum based optimizers are central to a wide range of machine learning applications. These typically rely on an Exponential Moving Average (EMA) of gradients, which decays exponentially the present contribution of older gradients. This accounts for gradients being local linear approximations which lose their relevance as the iterate moves along the loss landscape. This work questions the use of a single EMA to accumulate past gradients and empirically demonstrates how this choice can be sub-optimal: a single EMA cannot simultaneously give a high weight to the immediate past, and a non-negligible weight to older gradients. Building on this observation, we propose AdEMAMix, a simple modification of the Adam optimizer with a mixture of two EMAs to better take advantage of past gradients. Our experiments on language modeling and image classification show -- quite surprisingly -- that gradients can stay relevant for tens of thousands of steps. They help to converge faster, and often to lower minima: e.g., a $1.3$B parameter AdEMAMix LLM trained on $101$B tokens performs comparably to an AdamW model trained on $197$B tokens ($+95\\%$). Moreover, our method significantly slows-down model forgetting during training. Our work motivates further exploration of different types of functions to leverage past gradients, beyond EMAs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [In-context Learning on Fineweb-Edu 16.8B tokens](<https://www.sota2.com/research/sota/in-context-learning-on-fineweb-edu-16-8b-tokens.md>): In-Context Learning; Fineweb-Edu 16.8B tokens; ARC-c Accuracy: 35.41.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.03137>)

[PDF](<https://arxiv.org/pdf/2409.03137.pdf>)
