# Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

Source: [Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation](<https://www.sota2.com/research/paper/6902f0f33ef7ccd74e5e42a5>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:03:19.763Z

Authors: Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang, Kate Sanders, Eugene Yang, Chihsheng Jin, Benjamin Van Durme

Published: 2025-10-28T18:21:19.000Z

## Abstract

We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a prevalent source of information online, it is essential for RAG systems to integrate information from these sources into generation. However, existing evaluations for RAG are text-centric, limiting their applicability to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE as well as multimodal variants of three prominent text-based RAG metrics -- ALCE, ARGUE, and RAGAS -- demonstrating the limitations of text-centric work and laying the groundwork for automatic evaluation. We release open-source implementations and outline evaluation methods for multimodal RAG.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multimodal RAG Evaluation on MAGMaR Extrinsic Quality Judgment (EQJ)](<https://www.sota2.com/research/sota/multimodal-rag-evaluation-on-magmar-extrinsic-quality-judgment-eqj.md>): Multimodal RAG Evaluation; MAGMaR Extrinsic Quality Judgment (EQJ); Information Preservation (Retrieval): 65.

- [Citation Quality Evaluation on WikiVideo 1.0 (test)](<https://www.sota2.com/research/sota/citation-quality-evaluation-on-wikivideo-1-0-test.md>): Citation Quality Evaluation; WikiVideo 1.0 (test); CITEP R: not recorded.

- [Extrinsic Quality Judgment on WikiVideo](<https://www.sota2.com/research/sota/extrinsic-quality-judgment-on-wikivideo.md>): Extrinsic Quality Judgment; WikiVideo; INFOP (Reference) Score: not recorded.

- [Intrinsic Claim Judgment on WikiVideo](<https://www.sota2.com/research/sota/intrinsic-claim-judgment-on-wikivideo.md>): Intrinsic Claim Judgment; WikiVideo; INFOP (Reference): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.24870>)

[PDF](<https://arxiv.org/pdf/2510.24870.pdf>)
