# Representation Collapsing Problems in Vector Quantization

Source: [Representation Collapsing Problems in Vector Quantization](<https://www.sota2.com/research/paper/68fa75d61e209131cea6a24a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:11.279Z

Authors: Wenhao Zhao, Qiran Zou, Rushi Shah, Dianbo Liu

Published: 2024-11-25T16:32:29.000Z

## Abstract

Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we investigate representation collapse in vector quantization - a critical degradation where codebook tokens or latent embeddings lose their discriminative power by converging to a limited subset of values. This collapse fundamentally compromises the model's ability to capture diverse data patterns. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that restricted initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Codebook utilization on WikiCS](<https://www.sota2.com/research/sota/codebook-utilization-on-wikics.md>): Codebook utilization; wikiCS; Perplexity: 3.1.

- [Graph Representation Learning on ogbn-products (test)](<https://www.sota2.com/research/sota/graph-representation-learning-on-ogbn-products-test.md>): Graph Representation Learning; ogbn-products (test); Perplexity: 8.79.

- [Graph Representation Learning on ogbn-proteins (test)](<https://www.sota2.com/research/sota/graph-representation-learning-on-ogbn-proteins-test.md>): Graph Representation Learning; ogbn-proteins (test); Perplexity: 8.47.

- [Codebook utilization on Cora](<https://www.sota2.com/research/sota/codebook-utilization-on-cora.md>): Codebook utilization; Cora; Perplexity: 60.57.

- [Codebook utilization on PubMed](<https://www.sota2.com/research/sota/codebook-utilization-on-pubmed.md>): Codebook utilization; Pubmed; Perplexity: 6.17.

- [Codebook utilization on Photo](<https://www.sota2.com/research/sota/codebook-utilization-on-photo.md>): Codebook utilization; Photo; Perplexity: 3.78.

- [Codebook utilization on Computer](<https://www.sota2.com/research/sota/codebook-utilization-on-computer.md>): Codebook utilization; Computer; Perplexity: 2.94.

- [Codebook utilization on Roman](<https://www.sota2.com/research/sota/codebook-utilization-on-roman.md>): Codebook utilization; Roman; Perplexity: 14.49.

- [Graph Representation Learning on ogbn-arxiv (test)](<https://www.sota2.com/research/sota/graph-representation-learning-on-ogbn-arxiv-test.md>): Graph Representation Learning; ogbn-arxiv (test); Perplexity: 7.85.

- [Codebook utilization on Ratings](<https://www.sota2.com/research/sota/codebook-utilization-on-ratings.md>): Codebook utilization; Ratings; Perplexity: 37.65.

- [Codebook utilization on Questions](<https://www.sota2.com/research/sota/codebook-utilization-on-questions.md>): Codebook utilization; questions; Perplexity: 58.99.

- [Codebook utilization on Citeseer](<https://www.sota2.com/research/sota/codebook-utilization-on-citeseer.md>): Codebook utilization; Citeseer; Perplexity: 138.98.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.16550>)

[PDF](<https://arxiv.org/pdf/2411.16550.pdf>)
