# Mirror Descent View for Neural Network Quantization

Source: [Mirror Descent View for Neural Network Quantization](<https://www.sota2.com/research/paper/68fa69731e209131cea2226b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:00:47.140Z

Authors: Thalaiyasingam Ajanthan, Kartik Gupta, Philip H. S. Torr, Richard Hartley, Puneet K. Dokania

Published: 2019-10-18T03:19:21.000Z

## Abstract

Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity. It is usually formulated as a constrained optimization problem and optimized via a modified version of gradient descent. In this work, by interpreting the continuous parameters (unconstrained) as the dual of the quantized ones, we introduce a Mirror Descent (MD) framework for NN quantization. Specifically, we provide conditions on the projections (i.e., mapping from continuous to quantized ones) which would enable us to derive valid mirror maps and in turn the respective MD updates. Furthermore, we present a numerically stable implementation of MD that requires storing an additional set of auxiliary variables (unconstrained), and show that it is strikingly analogous to the Straight Through Estimator (STE) based method which is typically viewed as a "trick" to avoid vanishing gradients issue. Our experiments on CIFAR-10/100, TinyImageNet, and ImageNet classification datasets with VGG-16, ResNet-18, and MobileNetV2 architectures show that our MD variants obtain quantized networks with state-of-the-art performance. Code is available at https://github.com/kartikgupta-at-anu/md-bnn.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on Tiny-ImageNet](<https://www.sota2.com/research/sota/image-classification-on-tiny-imagenet.md>): Image Classification; Tiny-ImageNet; Accuracy: 58.35.

- [Image Classification on CIFAR-10](<https://www.sota2.com/research/sota/image-classification-on-cifar-10.md>): Image Classification; CIFAR-10; Accuracy: 94.84.

- [Image Classification on CIFAR-100](<https://www.sota2.com/research/sota/image-classification-on-cifar-100.md>): Image Classification; CIFAR-100; Accuracy: 76.31.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1910.08237>)

[PDF](<https://arxiv.org/pdf/1910.08237.pdf>)
