# 8-bit Numerical Formats for Deep Neural Networks

Source: [8-bit Numerical Formats for Deep Neural Networks](<https://www.sota2.com/research/paper/68fa6cb41e209131cea355ef>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:16:46.056Z

Authors: Badreddine Noune, Philip Jones, Daniel Justus, Dominic Masters, Carlo Luschi

Published: 2022-06-06T21:31:32.000Z

## Abstract

Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2206.02915>)

[PDF](<https://arxiv.org/pdf/2206.02915.pdf>)