# LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate Source: [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:34:52.943Z Authors: Anthony Fuller, Daniel G. Kyrollos, Yousef Yassin, James R. Green Published: 2024-05-22T20:47:25.000Z ## Abstract High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectively leverage larger images without finetuning -- ViTs poorly extrapolate to more patches at test time, although transformers offer sequence length flexibility. We attribute this shortcoming to the current patch position encoding methods, which create a distribution shift when extrapolating. We propose a drop-in replacement for the position encoding of plain ViTs that restricts attention heads to fixed fields of view, pointed in different directions, using 2D attention masks. Our novel method, called LookHere, provides translation-equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that LookHere improves performance on classification (avg. 1.6%), against adversarial attack (avg. 5.4%), and decreases calibration error (avg. 1.5%) -- on ImageNet without extrapolation. With extrapolation, LookHere outperforms the current SoTA position encoding method, 2D-RoPE, by 21.7% on ImageNet when trained at $224^2$ px and tested at $1024^2$ px. Additionally, we release a high-resolution test set to improve the evaluation of high-resolution image classifiers, called ImageNet-HR. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Object Detection on COCO (mAP)](): Object Detection; COCO; mAP: 37.3. - [Image Classification on ImageNet-1K (test) (Standard Accuracy)](): Image Classification; ImageNet 1k (test); Top-1 Accuracy: 81.4. - [Image Classification on ImageNet-1K (Top-1 Accuracy)](): Image Classification; ImageNet-1K; Top-1 Acc: 80.5. - [Image Classification on ImageNet-1K (val) (Accuracy)](): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 81.4. ## Original sources and implementation [Original paper]() [PDF]()