# Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice Source: [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:01:55.246Z Authors: Peihao Wang, Wenqing Zheng, Tianlong Chen, Zhangyang Wang Published: 2022-03-09T23:55:24.000Z ## Abstract Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, a rigorous framework studying on this scalability issue remains elusive. In this paper, we first establish a rigorous theory framework to analyze ViT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, which indicates when ViT scales up its depth, excessive low-pass filtering will cause feature maps to only preserve their Direct-Current (DC) component. We then propose two straightforward yet effective techniques to mitigate the undesirable low-pass limitation. The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix. The second technique, termed FeatScale, re-weights feature maps on separate frequency bands to amplify the high-frequency signals. Both techniques are efficient and hyperparameter-free, while effectively overcoming relevant ViT training artifacts such as attention collapse and patch uniformity. By seamlessly plugging in our techniques to multiple ViT variants, we demonstrate that they consistently help ViTs benefit from deeper architectures, bringing up to 1.1% performance gains "for free" (e.g., with little parameter overhead). We publicly release our codes and pre-trained models at https://github.com/VITA-Group/ViT-Anti-Oversmoothing. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Object Classification on ImageNet (val) (Top-1 & Top-5 Acc)](): Object Classification; ImageNet (val); Top-1 Accuracy: 72.346. - [Image Classification on ImageNet-1K (test) (Standard Accuracy)](): Image Classification; ImageNet 1k (test); Top-1 Accuracy: 81.3. - [Image Classification on ImageNet-1k](): Image Classification; ImageNet-1K; Top-1 Acc: 81.3. - [Image Classification on ImageNet 1k (val)](): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 80.9. ## Original sources and implementation [Original paper]() [PDF]()