# Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention

Source: [Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention](<https://www.sota2.com/research/paper/68fa6f081e209131cea43703>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:11:39.114Z

Authors: Mingyu Ding, Yikang Shen, Lijie Fan, Zhenfang Chen, Zitian Chen, Ping Luo, Joshua B. Tenenbaum, Chuang Gan

Published: 2023-04-06T17:59:26.000Z

## Abstract

Humans possess a versatile mechanism for extracting structured representations of our visual world. When looking at an image, we can decompose the scene into entities and their parts as well as obtain the dependencies between them. To mimic such capability, we propose Visual Dependency Transformers (DependencyViT) that can induce visual dependencies without any labels. We achieve that with a novel neural operator called \\emph{reversed attention} that can naturally capture long-range visual dependencies between image patches. Specifically, we formulate it as a dependency graph where a child token in reversed attention is trained to attend to its parent tokens and send information following a normalized probability distribution rather than gathering information in conventional self-attention. With such a design, hierarchies naturally emerge from reversed attention layers, and a dependency tree is progressively induced from leaf nodes to the root node unsupervisedly.   DependencyViT offers several appealing benefits. (i) Entities and their parts in an image are represented by different subtrees, enabling part partitioning from dependencies; (ii) Dynamic visual pooling is made possible. The leaf nodes which rarely send messages can be pruned without hindering the model performance, based on which we propose the lightweight DependencyViT-Lite to reduce the computational and memory footprints; (iii) DependencyViT works well on both self- and weakly-supervised pretraining paradigms on ImageNet, and demonstrates its effectiveness on 8 datasets and 5 tasks, such as unsupervised part and saliency segmentation, recognition, and detection.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Unsupervised Image Saliency Detection on ECSSD 55 (test)](<https://www.sota2.com/research/sota/unsupervised-image-saliency-detection-on-ecssd-55-test.md>): Unsupervised Image Saliency Detection; ECSSD 55 (test); Accuracy: 78.4.

- [Unsupervised Image Saliency Detection on OMRON 84 (test)](<https://www.sota2.com/research/sota/unsupervised-image-saliency-detection-on-omron-84-test.md>): Unsupervised Image Saliency Detection; OMRON 84 (test); Accuracy: 67.2.

- [Unsupervised Image Saliency Detection on DUTS 71 (test)](<https://www.sota2.com/research/sota/unsupervised-image-saliency-detection-on-duts-71-test.md>): Unsupervised Image Saliency Detection; DUTS 71 (test); Acc: 73.2.

- [Image Classification on ImageNet-1K 1.0 (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-val.md>): Image Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy: 82.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2304.03282>)

[PDF](<https://arxiv.org/pdf/2304.03282.pdf>)
