# ClusterFormer: Clustering As A Universal Visual Learner

Source: [ClusterFormer: Clustering As A Universal Visual Learner](<https://www.sota2.com/research/paper/68fa70d11e209131cea4d668>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:50:52.513Z

Authors: James C. Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, Dongfang Liu

Published: 2023-09-22T22:12:30.000Z

## Abstract

This paper presents CLUSTERFORMER, a universal vision model that is based on the CLUSTERing paradigm with TransFORMER. It comprises two novel designs: 1. recurrent cross-attention clustering, which reformulates the cross-attention mechanism in Transformer and enables recursive updates of cluster centers to facilitate strong representation learning; and 2. feature dispatching, which uses the updated cluster centers to redistribute image features through similarity-based metrics, resulting in a transparent pipeline. This elegant design streamlines an explainable and transferable workflow, capable of tackling heterogeneous vision tasks (i.e., image classification, object detection, and image segmentation) with varying levels of clustering granularity (i.e., image-, box-, and pixel-level). Empirical results demonstrate that CLUSTERFORMER outperforms various well-known specialized architectures, achieving 83.41% top-1 acc. over ImageNet-1K for image classification, 54.2% and 47.0% mAP over MS COCO for object detection and instance segmentation, 52.4% mIoU over ADE20K for semantic segmentation, and 55.8% PQ over COCO Panoptic for panoptic segmentation. For its efficacy, we hope our work can catalyze a paradigm shift in universal models in computer vision.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet Mini](<https://www.sota2.com/research/sota/image-classification-on-imagenet-mini.md>): Image Classification; ImageNet mini; Top-1 Accuracy: 73.99.

- [Image Classification on CIFAR-100 (val)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-val.md>): Image Classification; CIFAR-100 (val); Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2309.13196>)

[PDF](<https://arxiv.org/pdf/2309.13196.pdf>)