# Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

Source: [Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models](<https://www.sota2.com/research/paper/6a167ad5dbd7150506f802bb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-08T05:00:28.404Z

Authors: Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao, Yiming Du, Haobo Li, Xiyu Ren, Ginny Wong, Simon See, Lishu Luo, Haodong Duan, Pasquale Minervini, Yangqiu Song

Published: 2026-05-26T16:24:29.000Z

## Abstract

Large vision-language models increasingly rely on long-context modeling to reason over documents, hour-level videos, and long-horizon agent trajectories, requiring them to locate relevant evidence across interleaved text and images. Prior work has studied this behavior using retrieval heads in large language models, but its copy-based criterion does not directly apply when evidence appears in images. We introduce a multimodal retrieval head detection method that scores attention from question tokens to textual or visual evidence. With this method, we show that multimodal retrieval heads are sparse, intrinsic, and causally important: only 4.4-10.2% of attention heads account for 50% of the positive retrieval-score mass, and masking the top-5% selected heads drops MMLongBench-Doc from 48.2% to 5.7% and SlideVQA from 71.2% to 8.9%, while random-head masking is far less damaging. Further analysis shows that these heads are partly shared across modalities yet remain dynamic within each modality, with image retrieval heads changing more than text retrieval heads as context length and haystack modality change. Without further training, we find that these heads can also be used directly to rank visually rich documents: on MMDocIR, Qwen3-VL-8B selected-head scoring improves Recall@1 by 7.7/7.4 macro/micro points for page retrieval and 6.3/6.8 points for layout retrieval over the strongest reported baseline.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Page Retrieval on MMDocIR 1.0 (full)](<https://www.sota2.com/research/sota/page-retrieval-on-mmdocir-1-0-full.md>): Page Retrieval; MMDocIR 1.0 (full); Macro R@1: 64.7.

- [Layout Retrieval on MMDocIR 1.0 (full)](<https://www.sota2.com/research/sota/layout-retrieval-on-mmdocir-1-0-full.md>): Layout Retrieval; MMDocIR 1.0 (full); Macro Recall@1: 39.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.27243>)

[PDF](<https://arxiv.org/pdf/2605.27243.pdf>)
