# Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping

Source: [Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping](<https://www.sota2.com/research/paper/68fa68d81e209131cea1ea18>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:47:52.778Z

Authors: Jeffrey Fong, Siwei Chen, Kaiqi Chen

Published: 2020-11-27T07:20:08.000Z

## Abstract

Training neural networks with large batch is of fundamental significance to deep learning. Large batch training remarkably reduces the amount of training time but has difficulties in maintaining accuracy. Recent works have put forward optimization methods such as LARS and LAMB to tackle this issue through adaptive layer-wise optimization using trust ratios. Though prevailing, such methods are observed to still suffer from unstable and extreme trust ratios which degrades performance. In this paper, we propose a new variant of LAMB, called LAMBC, which employs trust ratio clipping to stabilize its magnitude and prevent extreme values. We conducted experiments on image classification tasks such as ImageNet and CIFAR-10 and our empirical results demonstrate promising improvements across different batch sizes.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2011.13584>)

[PDF](<https://arxiv.org/pdf/2011.13584.pdf>)