# EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks

Source: [EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks](<https://www.sota2.com/research/paper/68fa63d41e209131cea0112d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T02:51:07.227Z

Authors: Xuanyi Dong, Guoliang Kang, Kun Zhan, Yi Yang

Published: 2017-09-22T08:31:45.000Z

## Abstract

For most state-of-the-art architectures, Rectified Linear Unit (ReLU) becomes a standard component accompanied with each layer. Although ReLU can ease the network training to an extent, the character of blocking negative values may suppress the propagation of useful information and leads to the difficulty of optimizing very deep Convolutional Neural Networks (CNNs). Moreover, stacking layers with nonlinear activations is hard to approximate the intrinsic linear transformations between feature representations.   In this paper, we investigate the effect of erasing ReLUs of certain layers and apply it to various representative architectures following deterministic rules. It can ease the optimization and improve the generalization performance for very deep CNN models. We find two key factors being essential to the performance improvement: 1) the location where ReLU should be erased inside the basic module; 2) the proportion of basic modules to erase ReLU; We show that erasing the last ReLU layer of all basic modules in a network usually yields improved performance. In experiments, our approach successfully improves the performance of various representative architectures, and we report the improved results on SVHN, CIFAR-10/100, and ImageNet. Moreover, we achieve competitive single-model performance on CIFAR-100 with 16.53% error rate compared to state-of-the-art.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet full (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-full-val.md>): Image Classification; ImageNet full (val); Top-1 Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1709.07634>)

[PDF](<https://arxiv.org/pdf/1709.07634.pdf>)