# Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks

Source: [Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks](<https://www.sota2.com/research/paper/68fa6f581e209131cea4526c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:18:28.677Z

Authors: Minyoung Huh, Brian Cheung, Pulkit Agrawal, Phillip Isola

Published: 2023-05-15T17:56:36.000Z

## Abstract

This work examines the challenges of training neural networks using vector quantization using straight-through estimation. We find that a primary cause of training instability is the discrepancy between the model embedding and the code-vector distribution. We identify the factors that contribute to this issue, including the codebook gradient sparsity and the asymmetric nature of the commitment loss, which leads to misaligned code-vector assignments. We propose to address this issue via affine re-parameterization of the code vectors. Additionally, we introduce an alternating optimization to reduce the gradient error introduced by the straight-through estimation. Moreover, we propose an improvement to the commitment loss to ensure better alignment between the codebook representation and the model embedding. These optimization methods improve the mathematical approximation of the straight-through estimation and, ultimately, the model performance. We demonstrate the effectiveness of our methods on several common model architectures, such as AlexNet, ResNet, and ViT, across various tasks, including image classification and generative modeling.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Codebook utilization on Citeseer](<https://www.sota2.com/research/sota/codebook-utilization-on-citeseer.md>): Codebook utilization; Citeseer; Perplexity: 9.03.

- [Codebook utilization on Cora](<https://www.sota2.com/research/sota/codebook-utilization-on-cora.md>): Codebook utilization; Cora; Perplexity: 75.32.

- [Graph Representation Learning on ogbn-arxiv (test)](<https://www.sota2.com/research/sota/graph-representation-learning-on-ogbn-arxiv-test.md>): Graph Representation Learning; ogbn-arxiv (test); Perplexity: 52.39.

- [Graph Representation Learning on ogbn-proteins (test)](<https://www.sota2.com/research/sota/graph-representation-learning-on-ogbn-proteins-test.md>): Graph Representation Learning; ogbn-proteins (test); Perplexity: 49.32.

- [Codebook utilization on WikiCS](<https://www.sota2.com/research/sota/codebook-utilization-on-wikics.md>): Codebook utilization; wikiCS; Perplexity: 83.55.

- [Codebook utilization on Ratings](<https://www.sota2.com/research/sota/codebook-utilization-on-ratings.md>): Codebook utilization; Ratings; Perplexity: 73.82.

- [Codebook utilization on Questions](<https://www.sota2.com/research/sota/codebook-utilization-on-questions.md>): Codebook utilization; questions; Perplexity: 66.57.

- [Graph Representation Learning on ogbn-products (test)](<https://www.sota2.com/research/sota/graph-representation-learning-on-ogbn-products-test.md>): Graph Representation Learning; ogbn-products (test); Perplexity: 67.71.

- [Codebook utilization on PubMed](<https://www.sota2.com/research/sota/codebook-utilization-on-pubmed.md>): Codebook utilization; Pubmed; Perplexity: 126.55.

- [Codebook utilization on Photo](<https://www.sota2.com/research/sota/codebook-utilization-on-photo.md>): Codebook utilization; Photo; Perplexity: 54.95.

- [Codebook utilization on Computer](<https://www.sota2.com/research/sota/codebook-utilization-on-computer.md>): Codebook utilization; Computer; Perplexity: 59.33.

- [Codebook utilization on Roman](<https://www.sota2.com/research/sota/codebook-utilization-on-roman.md>): Codebook utilization; Roman; Perplexity: 118.46.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.08842>)

[PDF](<https://arxiv.org/pdf/2305.08842.pdf>)
