# Probabilistic Weight Fixing: Large-scale training of neural network weight uncertainties for quantization

Source: [Probabilistic Weight Fixing: Large-scale training of neural network weight uncertainties for quantization](<https://www.sota2.com/research/paper/68fa70cd1e209131cea4d41d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:34:17.040Z

Authors: Christopher Subia-Waud, Srinandan Dasmahapatra

Published: 2023-09-24T08:04:28.000Z

## Abstract

Weight-sharing quantization has emerged as a technique to reduce energy expenditure during inference in large neural networks by constraining their weights to a limited set of values. However, existing methods for weight-sharing quantization often make assumptions about the treatment of weights based on value alone that neglect the unique role weight position plays. This paper proposes a probabilistic framework based on Bayesian neural networks (BNNs) and a variational relaxation to identify which weights can be moved to which cluster centre and to what degree based on their individual position-specific learned uncertainty distributions. We introduce a new initialisation setting and a regularisation term which allow for the training of BNNs under complex dataset-model combinations. By leveraging the flexibility of weight values captured through a probability distribution, we enhance noise resilience and downstream compressibility. Our iterative clustering procedure demonstrates superior compressibility and higher accuracy compared to state-of-the-art methods on both ResNet models and the more complex transformer-based architectures. In particular, our method outperforms the state-of-the-art quantization method top-1 accuracy by 1.6% on ImageNet using DeiT-Tiny, with its 5 million+ weights now represented by only 296 unique values.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (val) (Top-1, Params, Entropy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-top-1-params-entropy.md>): Image Classification; ImageNet (val); Top-1 Acc: 78.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2309.13575>)

[PDF](<https://arxiv.org/pdf/2309.13575.pdf>)

[Implementation](<https://github.com/subiawaud/PWFN>)