# Releasing Inequality Phenomenon in $\\ell\_{\\infty}$-norm Adversarial Training via Input Gradient Distillation

Source: [Releasing Inequality Phenomenon in $\\ell\_{\\infty}$-norm Adversarial Training via Input Gradient Distillation](<https://www.sota2.com/research/paper/68fa79a51e209131cea7ea6a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:02:45.345Z

Authors: Junxi Chen, Junhao Dong, Xiaohua Xie, Jianhuang Lai

Published: 2023-05-16T09:23:42.000Z

## Abstract

Adversarial training (AT) is considered the most effective defense against adversarial attacks. However, a recent study revealed that \\(\\ell\_{\\infty}\\)-norm adversarial training (\\(\\ell\_{\\infty}\\)-AT) will also induce unevenly distributed input gradients, which is called the inequality phenomenon. This phenomenon makes the \\(\\ell\_{\\infty}\\)-norm adversarially trained model more vulnerable than the standard-trained model when high-attribution or randomly selected pixels are perturbed, enabling robust and practical black-box attacks against \\(\\ell\_{\\infty}\\)-adversarially trained models. In this paper, we propose a simple yet effective method called Input Gradient Distillation (IGD) to release the inequality phenomenon in $\\ell\_{\\infty}$-AT. IGD distills the standard-trained teacher model's equal decision pattern into the $\\ell\_{\\infty}$-adversarially trained student model by aligning input gradients of the student model and the standard-trained model with the Cosine Similarity. Experiments show that IGD can mitigate the inequality phenomenon and its threats while preserving adversarial robustness. Compared to vanilla $\\ell\_{\\infty}$-AT, IGD reduces error rates against inductive noise, inductive occlusion, random noise, and noisy images in ImageNet-C by up to 60\\%, 16\\%, 50\\%, and 21\\%, respectively. Other than empirical experiments, we also conduct a theoretical analysis to explain why releasing the inequality phenomenon can improve such robustness and discuss why the severity of the inequality phenomenon varies according to the dataset's image resolution. Our code is available at https://github.com/fhdnskfbeuv/Inuput-Gradient-Distillation

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.09305>)

[PDF](<https://arxiv.org/pdf/2305.09305.pdf>)
