# LocalNewton: Reducing Communication Bottleneck for Distributed Learning

Source: [LocalNewton: Reducing Communication Bottleneck for Distributed Learning](<https://www.sota2.com/research/paper/68fa69f51e209131cea25407>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:47.876Z

Authors: Vipul Gupta, Avishek Ghosh, Michal Derezinski, Rajiv Khanna, Kannan Ramchandran, Michael Mahoney

Published: 2021-05-16T00:15:08.000Z

## Abstract

To address the communication bottleneck problem in distributed optimization within a master-worker framework, we propose LocalNewton, a distributed second-order algorithm with local averaging. In LocalNewton, the worker machines update their model in every iteration by finding a suitable second-order descent direction using only the data and model stored in their own local memory. We let the workers run multiple such iterations locally and communicate the models to the master node only once every few (say L) iterations. LocalNewton is highly practical since it requires only one hyperparameter, the number L of local iterations. We use novel matrix concentration-based techniques to obtain theoretical guarantees for LocalNewton, and we validate them with detailed empirical evaluation. To enhance practicability, we devise an adaptive scheme to choose L, and we show that this reduces the number of local iterations in worker machines between two model synchronizations as the training proceeds, successively refining the model quality at the master. Via extensive experiments using several real-world datasets with AWS Lambda workers and an AWS EC2 master, we show that LocalNewton requires fewer than 60% of the communication rounds (between master and workers) and less than 40% of the end-to-end running time, compared to state-of-the-art algorithms, to reach the same training~loss.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (Robustness under Dirichlet and Pathological Corruption)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-robustness-under-dirichlet-and-pathological-corruption.md>): Image Classification; CIFAR-10; Acc (Dirichlet, alpha=0.1): 50.9.

- [Image Classification on EMNIST (Robustness Metrics)](<https://www.sota2.com/research/sota/image-classification-on-emnist-robustness-metrics.md>): Image Classification; EMNIST; Accuracy (Dirichlet, alpha=0.1): 80.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2105.07320>)

[PDF](<https://arxiv.org/pdf/2105.07320.pdf>)