# Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning

Source: [Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning](<https://www.sota2.com/research/paper/69ddc9e97d0172140ec79820>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-10T05:00:46.690Z

Authors: Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, Zheng Zhang

Published: 2026-04-11T00:27:40.000Z

## Abstract

Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, the orthogonalization quality of Muon hinges on the number of Newton--Schulz (NS) iterations performed, which poses efficiency challenges due to its non-trivial computation and communication cost. We propose Muon$^2$, an extension of Muon, to improve both quality and efficiency by applying Adam-style adaptive second-moment preconditioning before orthogonalization. Our key insight is that the core challenge of polar approximation in Muon lies in the ill-conditioned momentum matrix, of which the spectrum is substantially improved by Muon$^2$, leading to faster convergence toward a practically sufficient orthogonalization. We further characterize the practical orthogonalization quality via directional alignment, under which Muon$^2$ demonstrates dramatic improvement over Muon at each polar step. Across GPT, LLaMA, and Mixture-of-Experts pre-training experiments up to 13B parameters, Muon$^2$ (and its memory-efficient variant Muon$^2$-F that preserves most of its benefits) consistently outperforms Muon and its variants while reducing NS iterations by 40%, and saves up to 1/4 training time over Muon when achieving the same loss.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on GPT Small (val)](<https://www.sota2.com/research/sota/language-modeling-on-gpt-small-val.md>): Language Modeling; GPT Small (val); Validation Perplexity: 27.95.

- [Language Modeling on GPT-Base (val)](<https://www.sota2.com/research/sota/language-modeling-on-gpt-base-val.md>): Language Modeling; GPT-Base (val); Validation Perplexity: 19.96.

- [Language Modeling on C4 (val)](<https://www.sota2.com/research/sota/language-modeling-on-c4-val.md>): Language Modeling; C4 (val); PPL: 10.21.

- [Language Modeling on FineWeb (val)](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md>): Language Modeling; FineWeb (val); Validation Loss: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.09967>)

[PDF](<https://arxiv.org/pdf/2604.09967.pdf>)
