# Enhancing LLM Training via Spectral Clipping

Source: [Enhancing LLM Training via Spectral Clipping](<https://www.sota2.com/research/paper/69b8e03d0636e5bf4afbed96>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-30T05:00:11.986Z

Authors: Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich

Published: 2026-03-15T10:16:40.000Z

## Abstract

While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights and gradients, leaving them vulnerable to two empirical issues in large language model (LLM) training: (i) the optimizer updates can have large spectral norms, potentially destabilizing training and degrading generalization; (ii) stochastic gradient noise can exhibit sparse spectral spikes, with a few dominant singular values much larger than the rest. We propose SPECTRA, a general framework addressing these by (i) post-spectral clipping of updates to enforce spectral-norm constraints (ii) optional pre-spectral clipping of gradients to suppress spectral noise spikes. We prove that post-clipping constitutes a Composite Frank-Wolfe method with spectral-norm constraints and weight regularization. We further analyze how pre-clipping mitigates sparse spectral spikes. We propose efficient soft spectral clipping via Newton-Schulz iterations, avoiding expensive SVD. Experiments on LLM pretraining show SPECTRA uniformly improves validation loss for various optimizers, including AdamW, Signum, Mars, and AdEMAMix, with the best-performing variants achieving state-of-the-art results. Models trained with SPECTRA exhibit smaller weight norms, confirming the link between spectral clipping and regularization.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [In-context Learning on Fineweb-Edu 16.8B tokens](<https://www.sota2.com/research/sota/in-context-learning-on-fineweb-edu-16-8b-tokens.md>): In-Context Learning; Fineweb-Edu 16.8B tokens; ARC-c Accuracy: 36.86.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.14315>)

[PDF](<https://arxiv.org/pdf/2603.14315.pdf>)
