# Distributed Low Precision Training Without Mixed Precision

Source: [Distributed Low Precision Training Without Mixed Precision](<https://www.sota2.com/research/paper/68fa66d11e209131cea128f4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:59:21.344Z

Authors: Zehua Cheng, Weiyang Wang, Yan Pan, Thomas Lukasiewicz

Published: 2019-11-18T00:56:52.000Z

## Abstract

Low precision training is one of the most popular strategies for deploying the deep model on limited hardware resources. Fixed point implementation of DCNs has the potential to alleviate complexities and facilitate potential deployment on embedded hardware. However, most low precision training solution is based on a mixed precision strategy. In this paper, we have presented an ablation study on different low precision training strategy and propose a solution for IEEE FP-16 format throughout the training process. We tested the ResNet50 on 128 GPU cluster on ImageNet-full dataset. We have viewed that it is not essential to use FP32 format to train the deep models. We have viewed that communication cost reduction, model compression, and large-scale distributed training are three coupled problems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1911.07384>)

[PDF](<https://arxiv.org/pdf/1911.07384.pdf>)