# Predicting LLM Compression Degradation from Spectral Statistics

Source: [Predicting LLM Compression Degradation from Spectral Statistics](<https://www.sota2.com/research/paper/69e704dc7d0172140ec7f07a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-22T05:24:29.470Z

Authors: Mingxue Xu

Published: 2026-04-20T11:01:19.000Z

## Abstract

Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We systematically analyze the Qwen3 and Gemma3 model families across four representative low-rank compression methods: vanilla SVD, two ASVD variants, and SVD-LLM. We find that stable rank and information density, measured in bits per parameter, dominate performance degradation. The interaction term $\\gamma \\cdot \\bar{\\rho}\_s$, defined as compression ratio times stable rank, is a robust predictor of accuracy degradation, achieving leave-one-out cross-validation Pearson correlations of $0.890$ for attention layers and $0.839$ for MLP layers. We provide theoretical intuition for why this predictor succeeds by connecting it to standard SVD truncation bounds and error composition mechanisms in transformer layers. These findings enable a predict-then-compress workflow: compute $\\gamma \\cdot \\bar{\\rho}\_s$ from weights, estimate degradation, and invest compute only in desirable configurations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Accuracy Prediction on LLM Compression Performance Accuracy](<https://www.sota2.com/research/sota/accuracy-prediction-on-llm-compression-performance-accuracy.md>): Accuracy Prediction; LLM Compression Performance Accuracy; LOO R Score: not recorded.

- [Perplexity Prediction on LLM Compression Performance Perplexity](<https://www.sota2.com/research/sota/perplexity-prediction-on-llm-compression-performance-perplexity.md>): Perplexity Prediction; LLM Compression Performance Perplexity; LOO R: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.18085>)

[PDF](<https://arxiv.org/pdf/2604.18085.pdf>)