# ESPACE: Dimensionality Reduction of Activations for Model Compression

Source: [ESPACE: Dimensionality Reduction of Activations for Model Compression](<https://www.sota2.com/research/paper/68fa75221e209131cea66070>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:57:28.687Z

Authors: Charbel Sakr, Brucek Khailany

Published: 2024-10-07T18:59:22.000Z

## Abstract

We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations. Unlike prior works on weight-centric tensor decomposition, ESPACE projects activations onto a pre-calibrated set of principal components. The activation-centrality of the approach enables retraining LLMs with no loss of expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix multiplication associativity. Theoretical results on the construction of projection matrices with optimal computational accuracy are provided. Experimentally, we find ESPACE enables 50% compression of GPT3, Llama2, and Nemotron4 models with small accuracy degradation, as low as a 0.18 perplexity increase on GPT3-22B. At lower compression rates of 20% to 40%, ESPACE drives GPT3 models to outperforming their baseline, by up to a 0.38 decrease in perplexity for GPT3-8B. ESPACE also reduces GEMM execution time and prefill inference latency on existing hardware. Comparison with related works on compressing Llama2-7B via matrix factorization shows that ESPACE is a first step in advancing the state-of-the-art in tensor decomposition compression of LLMs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on WikiText-103](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-103.md>): Language Modeling; WikiText-103; PPL: 4.59.

- [Inference Efficiency on Model Profiling](<https://www.sota2.com/research/sota/inference-efficiency-on-model-profiling.md>): Inference Efficiency; Model Profiling; Total GEMM Latency (ms): 15.9.

- [Zero-shot Accuracy on Downstream Suite (BoolQ, HellaSwag, PIQA, RACE, WinoGrande)](<https://www.sota2.com/research/sota/zero-shot-accuracy-on-downstream-suite-boolq-hellaswag-piqa-race-winogrande.md>): Zero-shot Downstream Accuracy; Downstream Suite Zero-shot (BoolQ, HellaSwag, PIQA, RACE, WinoGrande); BoolQ Accuracy: 78.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2410.05437>)

[PDF](<https://arxiv.org/pdf/2410.05437.pdf>)