# Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers Source: [Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers]() Collection index: [llms.txt]() Data updated: 2026-05-19T05:00:13.117Z Authors: Hongyi Pan, Emadeldeen Hamdan, Xin Zhu, Ahmet Enis Cetin, Ulas Bagci Published: 2024-05-22T18:15:42.000Z ## Abstract Self-attention is central to the success of Transformer architectures; however, learning the query, key, and value projections from random initialization remains challenging and computationally expensive. In this paper, we propose two complementary methods that leverage the Discrete Cosine Transform (DCT) to enhance the efficiency and performance of Vision Transformers. First, we address the initialization problem by introducing a simple yet effective DCT-based initialization strategy for self-attention, where projection weights are initialized using DCT coefficients. This structure-preserving approach consistently improves classification accuracy on the CIFAR-10 and ImageNet-1K benchmarks. Second, we propose a DCT-based attention compression technique that exploits the decorrelation properties of the frequency domain. By observing that high-frequency DCT coefficients typically correspond to noise, we truncate high-frequency components of the input patches, thereby reducing the dimensionality of the query, key, and value projections without sacrificing accuracy. Experiments on Swin Transformer models demonstrate that the proposed compression method achieves a substantial reduction in computational overhead while maintaining comparable performance. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Instance Segmentation on COCO 2017](): Instance Segmentation; COCO 2017; APm: 45.1. - [Object Detection on COCO 2017](): Object Detection; COCO 2017; AP (Box): 52.4. ## Original sources and implementation [Original paper]() [PDF]()