# CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor

Source: [CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-17T05:00:21.724Z

Authors: Bishnu Dev, Sushil Bohara, Martin Tak\\'a\\v{c}, Samuel Horv\\'ath (1) \_\_INSTITUTION\_4\_\_ Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE)

Published: 2026-06-15T08:09:15.000Z

## Abstract

Muon is an optimizer that computes updates using the polar factor of the momentum matrix and has shown strong empirical performance across a range of training settings. A key component of Muon is the Newton-Schulz iteration used to compute this polar factor. Although this avoids the cost of an exact singular value decomposition, it remains expensive in practice because it is applied at every optimization step. At the same time, the momentum matrix changes smoothly over training, suggesting strong temporal correlation in the corresponding polar factors. In this paper, we exploit this structure and propose CacheMuon, a temporal preconditioning method that reuses information from previous optimization steps to approximate the polar factor at the current step. This reduces redundant orthogonalization computation across iterations. We analyze CacheMuon as an inexact Muon update, with error controlled by fresh-solver error and cache staleness. Empirically, CacheMuon provides a controllable quality-efficiency frontier: conservative thresholds closely match fresh Muon on language-model and vision training while reducing orthogonalization FLOPs, whereas more aggressive thresholds yield larger arithmetic savings at the cost of modest validation-quality degradation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on OpenWebText 1B tokens](<https://www.sota2.com/research/sota/language-modeling-on-openwebtext-1b-tokens.md>): Language Modeling; OpenWebText 1B tokens; Cumulative Orthogonalization Cost (PFLOPs): 26.67.

- [Image Classification on CIFAR-10 100 epochs](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-100-epochs.md>): Image Classification; CIFAR-10 100 epochs; Cumulative Orthogonalization Cost (PFLOPs): 1.98.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.16371>)

[PDF](<https://arxiv.org/pdf/2606.16371.pdf>)
