# ClusTR: Exploring Efficient Self-attention via Clustering for Vision Transformers Source: [ClusTR: Exploring Efficient Self-attention via Clustering for Vision Transformers]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:30:35.914Z Authors: Yutong Xie, Jianpeng Zhang, Yong Xia, Anton van den Hengel, Qi Wu Published: 2022-08-28T04:18:27.000Z ## Abstract Although Transformers have successfully transitioned from their language modelling origins to image-based applications, their quadratic computational complexity remains a challenge, particularly for dense prediction. In this paper we propose a content-based sparse attention method, as an alternative to dense self-attention, aiming to reduce the computation complexity while retaining the ability to model long-range dependencies. Specifically, we cluster and then aggregate key and value tokens, as a content-based method of reducing the total token count. The resulting clustered-token sequence retains the semantic diversity of the original signal, but can be processed at a lower computational cost. Besides, we further extend the clustering-guided attention from single-scale to multi-scale, which is conducive to dense prediction tasks. We label the proposed Transformer architecture ClusTR, and demonstrate that it achieves state-of-the-art performance on various vision tasks but at lower computational cost and with fewer parameters. For instance, our ClusTR small model with 22.7M parameters achieves 83.2\\% Top-1 accuracy on ImageNet. Source code and ImageNet models will be made publicly available. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()