# Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

Source: [Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training](<https://www.sota2.com/research/paper/68fa6b891e209131cea2ef4f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T05:04:56.791Z

Authors: Sheng Liu, Xiao Li, Yuexiang Zhai, Chong You, Zhihui Zhu, Carlos Fernandez-Granda, Qing Qu

Published: 2021-03-01T00:33:04.000Z

## Abstract

Normalization techniques have become a basic component in modern convolutional neural networks (ConvNets). In particular, many recent works demonstrate that promoting the orthogonality of the weights helps train deep models and improve robustness. For ConvNets, most existing methods are based on penalizing or normalizing weight matrices derived from concatenating or flattening the convolutional kernels. These methods often destroy or ignore the benign convolutional structure of the kernels; therefore, they are often expensive or impractical for deep ConvNets. In contrast, we introduce a simple and efficient "Convolutional Normalization" (ConvNorm) method that can fully exploit the convolutional structure in the Fourier domain and serve as a simple plug-and-play module to be conveniently incorporated into any ConvNets. Our method is inspired by recent work on preconditioning methods for convolutional sparse coding and can effectively promote each layer's channel-wise isometry. Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets. Applied to classification under noise corruptions and generative adversarial network (GAN), we show that the ConvNorm improves the robustness of common ConvNets such as ResNet and the performance of GAN. We verify our findings via numerical experiments on CIFAR and ImageNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Adversarial Robustness on CIFAR-10 (test)](<https://www.sota2.com/research/sota/adversarial-robustness-on-cifar-10-test.md>): Adversarial Robustness; CIFAR-10 (test); Attack Success Rate (ASR): 53.5.

- [Image Generation on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-generation-on-cifar-10-test.md>): Image Generation; CIFAR-10 (test); FID: 21.37.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: 93.31.

- [Image Classification on CIFAR-100 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test.md>): Image Classification; CIFAR-100 (test); Accuracy: 73.38.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2103.00673>)

[PDF](<https://arxiv.org/pdf/2103.00673.pdf>)

[Implementation](<https://github.com/shengliu66/ConvNorm>)
