# AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learning Source: [AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learning]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:59:03.375Z Authors: Daniel Coquelin, Katherina Fl\\"ugel, Marie Weiel, Nicholas Kiefer, Muhammed \\"Oz, Charlotte Debus, Achim Streit, Markus G\\"otz Published: 2024-05-02T07:49:28.000Z ## Abstract Communication bottlenecks severely hinder the scalability of distributed neural network training, particularly in high-performance computing (HPC) environments. We introduce AB-training, a novel data-parallel method that leverages low-rank representations and independent training groups to significantly reduce communication overhead. Our experiments demonstrate an average reduction in network traffic of approximately 70.31\\% across various scaling scenarios, increasing the training potential of communication-constrained systems and accelerating convergence at scale. AB-training also exhibits a pronounced regularization effect at smaller scales, leading to improved generalization while maintaining or even reducing training time. We achieve a remarkable 44.14 : 1 compression ratio on VGG16 trained on CIFAR-10 with minimal accuracy loss, and outperform traditional data parallel training by 1.55\\% on ResNet-50 trained on ImageNet-2012. While AB-training is promising, our findings also reveal that large batch effects persist even in low-rank regimes, underscoring the need for further research into optimized update mechanisms for massively distributed training. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()