# GradAug: A New Regularization Method for Deep Neural Networks

Source: [GradAug: A New Regularization Method for Deep Neural Networks](<https://www.sota2.com/research/paper/68fa687e1e209131cea1c529>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:38:00.142Z

Authors: Taojiannan Yang, Sijie Zhu, Chen Chen

Published: 2020-06-14T19:30:34.000Z

## Abstract

We propose a new regularization method to alleviate over-fitting in deep neural networks. The key idea is utilizing randomly transformed training samples to regularize a set of sub-networks, which are originated by sampling the width of the original network, in the training process. As such, the proposed method introduces self-guided disturbances to the raw gradients of the network and therefore is termed as Gradient Augmentation (GradAug). We demonstrate that GradAug can help the network learn well-generalized and more diverse representations. Moreover, it is easy to implement and can be applied to various structures and applications. GradAug improves ResNet-50 to 78.79% on ImageNet classification, which is a new state-of-the-art accuracy. By combining with CutMix, it further boosts the performance to 79.67%, which outperforms an ensemble of advanced training tricks. The generalization ability is evaluated on COCO object detection and instance segmentation where GradAug significantly surpasses other state-of-the-art methods. GradAug is also robust to image distortions and FGSM adversarial attacks and is highly effective in low data regimes. Code is available at https://github.com/taoyang1122/GradAug

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-100 WRN-28-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-wrn-28-10-test.md>): Image Classification; CIFAR-100 WRN-28-10 (test); Top-1 Accuracy: 84.05.

- [Image Classification on ImageNet-200](<https://www.sota2.com/research/sota/image-classification-on-imagenet-200.md>): Image Classification; ImageNet-200; Accuracy: 0.635.

- [Image Classification on CIFAR-10 (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-standard-accuracy.md>): Image Classification; CIFAR-10; Accuracy: 84.6.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: not recorded.

- [Image Classification on CIFAR-100 (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-standard-accuracy.md>): Image Classification; CIFAR-100; Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2006.07989>)

[PDF](<https://arxiv.org/pdf/2006.07989.pdf>)