# Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects

Source: [Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a00d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T08:08:08.607Z

Authors: Michael A. Lepori, Alexa R. Tartaglini, Wai Keen Vong, Thomas Serre, Brenden M. Lake, Ellie Pavlick

Published: 2024-06-22T22:43:10.000Z

## Abstract

Though vision transformers (ViTs) have achieved state-of-the-art performance in a variety of settings, they exhibit surprising failures when performing tasks involving visual relations. This begs the question: how do ViTs attempt to perform tasks that require computing visual relations between objects? Prior efforts to interpret ViTs tend to focus on characterizing relevant low-level visual features. In contrast, we adopt methods from mechanistic interpretability to study the higher-level visual algorithms that ViTs use to perform abstract visual reasoning. We present a case study of a fundamental, yet surprisingly difficult, relational reasoning task: judging whether two visual entities are the same or different. We find that pretrained ViTs fine-tuned on this task often exhibit two qualitatively different stages of processing despite having no obvious inductive biases to do so: 1) a perceptual stage wherein local object features are extracted and stored in a disentangled representation, and 2) a relational stage wherein object representations are compared. In the second stage, we find evidence that ViTs can learn to represent somewhat abstract visual relations, a capability that has long been considered out of reach for artificial neural networks. Finally, we demonstrate that failures at either stage can prevent a model from learning a generalizable solution to our fairly simple tasks. By understanding ViTs in terms of discrete processing stages, one can more precisely diagnose and rectify shortcomings of existing and future models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Discrimination task 256 shape-color combinations (train)](<https://www.sota2.com/research/sota/discrimination-task-256-shape-color-combinations-train.md>): Discrimination; Discrimination task 256 shape-color combinations (train); Train Accuracy: not recorded.

- [Discrimination on 256 shape-color combinations (test)](<https://www.sota2.com/research/sota/discrimination-on-256-shape-color-combinations-test.md>): Discrimination; Discrimination task 256 shape-color combinations (test); Test Accuracy: not recorded.

- [Discrimination on Photorealistic discrimination dataset (test)](<https://www.sota2.com/research/sota/discrimination-on-photorealistic-discrimination-dataset-test.md>): Discrimination; Photorealistic discrimination dataset (test); Realistic Acc: not recorded.

- [Relational Match-to-Sample (RMTS) on RMTS 256 shape-color combinations 1.0 (train)](<https://www.sota2.com/research/sota/relational-match-to-sample-rmts-on-rmts-256-shape-color-combinations-1-0-train.md>): Relational Match-to-Sample (RMTS); RMTS 256 shape-color combinations 1.0 (train); Accuracy: not recorded.

- [Relational Match-to-Sample on RMTS 256 shape-color combinations (test)](<https://www.sota2.com/research/sota/relational-match-to-sample-on-rmts-256-shape-color-combinations-test.md>): Relational Match-to-Sample (RMTS); RMTS 256 shape-color combinations 1.0 (test); Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2406.15955>)

[PDF](<https://arxiv.org/pdf/2406.15955.pdf>)
