# Enhancing Adversarial Defense by k-Winners-Take-All

Source: [Enhancing Adversarial Defense by k-Winners-Take-All](<https://www.sota2.com/research/paper/68fa66911e209131cea10f24>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:40.732Z

Authors: Chang Xiao, Peilin Zhong, Changxi Zheng

Published: 2019-05-25T03:36:40.000Z

## Abstract

We propose a simple change to existing neural network structures for better defending against gradient-based adversarial attacks. Instead of using popular activation functions (such as ReLU), we advocate the use of k-Winners-Take-All (k-WTA) activation, a C0 discontinuous function that purposely invalidates the neural network model's gradient at densely distributed input data points. The proposed k-WTA activation can be readily used in nearly all existing networks and training methods with no significant overhead. Our proposal is theoretically rationalized. We analyze why the discontinuities in k-WTA networks can largely prevent gradient-based search of adversarial examples and why they at the same time remain innocuous to the network training. This understanding is also empirically backed. We test k-WTA activation on various network structures optimized by a training method, be it adversarial training or not. In all cases, the robustness of k-WTA networks outperforms that of traditional networks under white-box attacks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Adversarial Robustness on CIFAR-10 (L-inf, eps=4/255)](<https://www.sota2.com/research/sota/adversarial-robustness-on-cifar-10-l-inf-eps-4-255.md>): Adversarial Robustness; CIFAR-10 L-infinity, epsilon=4/255; Robust Accuracy (AA): 64.91.

- [Adversarial Robustness on CIFAR-10 (L2, eps=8/255)](<https://www.sota2.com/research/sota/adversarial-robustness-on-cifar-10-l2-eps-8-255.md>): Adversarial Robustness; CIFAR-10 L2, epsilon=8/255; Robust Acc (AA): 5.4.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: 79.28.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1905.10510>)

[PDF](<https://arxiv.org/pdf/1905.10510.pdf>)