# The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

Source: [The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry](<https://www.sota2.com/research/paper/69f03f0755a3c32af89f9f94>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-29T05:00:05.003Z

Authors: Yi Liu

Published: 2026-04-03T08:58:53.000Z

## Abstract

We present the first systematic study of weight matrix singular value spectra \\emph{during} transformer pretraining, tracking full SVD decompositions of every weight matrix at 25-step intervals across three model scales (30M--285M parameters). We discover three phenomena: \\textbf{(1)~Transient Compression Waves:} stable rank compression propagates as a traveling wave from early to late layers, creating a dramatic gradient that peaks early then \\emph{reverses} -- late layers eventually over-compress past early layers. \\textbf{(2)~Persistent Spectral Gradients:} the power-law exponent~$\\alpha$ develops a permanent depth gradient forming a non-monotonic inverted-U in deeper models, with peaks shifting toward earlier layers as depth increases. \\textbf{(3)~Q/K--V Functional Asymmetry:} value/output projections compress uniformly while query/key projections carry the full depth-dependent dynamics. The dissociation between transient compression and persistent spectral shape reveals that \\emph{rank and spectral shape encode fundamentally different information about training}. We formalize this as a two-timescale dynamical model and derive scaling laws ($\\Delta\\alpha \\propto L^{0.26}$, $R^2{=}0.99$). We validate on nine models across three families (custom, GPT-2, Pythia; 30M--1B parameters; 8--36 layers), demonstrate that $\\alpha$ predicts layer importance ($\\rho{=}0.69$--$0.84$, $p{\<}0.02$), and show that spectral-guided pruning outperforms Last-N heuristics by $1.1{\\times}$--$3.6{\\times}$ across seven models in two families (GPT-2 124M--774M, Pythia 160M--1B), with worst-vs-best gaps up to $23.7{\\times}$ confirming the causal role of spectral structure.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on GPT-2](<https://www.sota2.com/research/sota/language-modeling-on-gpt-2.md>): Language Modeling; GPT-2; Delta PPL: 0.5.

- [Language Modeling on Pythia-160M](<https://www.sota2.com/research/sota/language-modeling-on-pythia-160m.md>): Language Modeling; Pythia-160M; Delta Perplexity (ΔPPL): 35.21.

- [Pruning on Pythia 1B (16L), 410M (24L) (val)](<https://www.sota2.com/research/sota/pruning-on-pythia-1b-16l-410m-24l-val.md>): Pruning; Pythia 1B (16L), 410M (24L) (val); ΔPPL: 5.6.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.22778>)

[PDF](<https://arxiv.org/pdf/2604.22778.pdf>)