# Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Source: [Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning](<https://www.sota2.com/research/paper/68fa6ca41e209131cea352bd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:15:46.115Z

Authors: Chong Ma, Lin Zhao, Yuzhong Chen, Lu Zhang, Zhenxiang Xiao, Haixing Dai, David Liu, Zihao Wu, Zhengliang Liu, Sheng Wang, Jiaxing Gao, Changhe Li, Xi Jiang, Tuo Zhang, Qian Wang, Dinggang Shen, Dajiang Zhu, Tianming Liu

Published: 2022-05-25T03:29:10.000Z

## Abstract

Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning the meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical imaging, where the clinical data (e.g., MR images with pathology) are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To address this problem, we propose to infuse human experts' intelligence and domain knowledge into the training of deep neural networks. The core idea is that we infuse the visual attention information from expert radiologists to proactively guide the deep model to focus on regions with potential pathology and avoid being trapped in learning harmful shortcuts. To do so, we propose a novel eye-gaze-guided vision transformer (EG-ViT) for diagnosis with limited medical image data. We mask the input image patches that are out of the radiologists' interest and add an additional residual connection in the last encoder layer of EG-ViT to maintain the correlations of all patches. The experiments on two public datasets of INbreast and SIIM-ACR demonstrate our EG-ViT model can effectively learn/transfer experts' domain knowledge and achieve much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the EG-ViT model's interpretability. In general, EG-ViT takes the advantages of both human expert's prior knowledge and the power of deep neural networks. This work opens new avenues for advancing current artificial intelligence paradigms by infusing human intelligence.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Classification on SIIM-ACR official (test)](<https://www.sota2.com/research/sota/classification-on-siim-acr-official-test.md>): Classification; SIIM-ACR official (test); Accuracy: 85.6.

- [Pairwise Perceived Cycling Safety Modeling on PCS Berlin (test)](<https://www.sota2.com/research/sota/pairwise-perceived-cycling-safety-modeling-on-pcs-berlin-test.md>): Pairwise Perceived Cycling Safety Modeling; PCS Berlin (test); Rank Accuracy: 74.41.

- [Classification on CXR-Gaze official (test)](<https://www.sota2.com/research/sota/classification-on-cxr-gaze-official-test.md>): Classification; CXR-Gaze official (test); AUC: 90.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2205.12466>)

[PDF](<https://arxiv.org/pdf/2205.12466.pdf>)
