# Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark

Source: [Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark](<https://www.sota2.com/research/paper/68fa77581e209131cea728c4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:15:51.701Z

Authors: Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, David M. Chan

Published: 2024-07-18T17:59:30.000Z

## Abstract

Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images. Recent advancements like long-context LMMs have allowed them to ingest larger, or even multiple, images. However, the ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering (MIQA), especially in real-world applications like photo album searches or satellite imagery analysis. In this work, we first assess the limitations of current benchmarks for long-context LMMs. We address these limitations by introducing a new vision-centric, long-context benchmark, "Visual Haystacks (VHs)". We comprehensively evaluate both open-source and proprietary models on VHs, and demonstrate that these models struggle when reasoning across potentially unrelated images, perform poorly on cross-image reasoning, as well as exhibit biases based on the placement of key information within the context window. Towards a solution, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU -- far surpassing the 1k-image limit of contemporary models. MIRAGE demonstrates up to 13% performance improvement over existing open-source LMMs on VHs, sets a new state-of-the-art on the RetVQA multi-image QA benchmark, and achieves competitive performance on single-image QA with state-of-the-art LMMs. Our dataset, model, and code are available at: https://visual-haystacks.github.io.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Question Answering on DocHaystack-1000](<https://www.sota2.com/research/sota/visual-question-answering-on-dochaystack-1000.md>): Visual Question Answering; DocHaystack-1000; Accuracy: 2.75.

- [Visual Question Answering on InfoHaystack-1000](<https://www.sota2.com/research/sota/visual-question-answering-on-infohaystack-1000.md>): Visual Question Answering; InfoHaystack-1000; Accuracy: 6.45.

- [Visual Question Answering on DocHaystack 100](<https://www.sota2.com/research/sota/visual-question-answering-on-dochaystack-100.md>): Visual Question Answering; DocHaystack 100; Accuracy: 0.0367.

- [Visual Question Answering on DocHaystack 200](<https://www.sota2.com/research/sota/visual-question-answering-on-dochaystack-200.md>): Visual Question Answering; DocHaystack 200; Accuracy: 3.67.

- [Visual Question Answering on InfoHaystack-100](<https://www.sota2.com/research/sota/visual-question-answering-on-infohaystack-100.md>): Visual Question Answering; InfoHaystack 100; Accuracy: 7.74.

- [Visual Question Answering on InfoHaystack-200](<https://www.sota2.com/research/sota/visual-question-answering-on-infohaystack-200.md>): Visual Question Answering; InfoHaystack 200; Accuracy: 7.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2407.13766>)

[PDF](<https://arxiv.org/pdf/2407.13766.pdf>)