# APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm

Source: [APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm](<https://www.sota2.com/research/paper/68fa68331e209131cea1ab7a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:32:00.581Z

Authors: Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari, Xiangru Lian, Ji Liu, Yuxiong He, Ce Zhang

Published: 2020-08-26T02:20:23.000Z

## Abstract

Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet. However, Adam is generally not compatible with information (gradient) compression technology. Therefore, the communication usually becomes the bottleneck for parallelizing Adam. In this paper, we propose a communication efficient {\\bf A}DAM {\\bf p}reconditioned {\\bf M}omentum SGD algorithm-- named APMSqueeze-- through an error compensated method compressing gradients. The proposed algorithm achieves a similar convergence efficiency to Adam in term of epochs, but significantly reduces the running time per epoch. In terms of end-to-end performance (including the full-precision pre-condition step), APMSqueeze is able to provide {sometimes by up to $2-10\\times$ speed-up depending on network bandwidth.} We also conduct theoretical analysis on the convergence and efficiency.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2008.11343>)

[PDF](<https://arxiv.org/pdf/2008.11343.pdf>)