# BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

Source: [BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights](<https://www.sota2.com/research/paper/68fa552b1e209131ce9919ae>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T13:24:40.416Z

Authors: Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, Jack Xin

Published: 2018-01-19T06:35:23.000Z

## Abstract

We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of \\emph{pseudo} quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights towards the quantized state by gradually increasing the regularization parameter. In the second phase, exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy over the state-of-the-art training methods. Finally, we prove the convergence of BinaryRelax under an approximate orthogonality condition.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: 90.65.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1801.06313>)

[PDF](<https://arxiv.org/pdf/1801.06313.pdf>)
