# MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation

Source: [MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation](<https://www.sota2.com/research/paper/68fa7bb61e209131cea8a163>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-15T05:00:04.560Z

Authors: Yuelyu Ji, Wuwei Lan, Patrick NG

Published: 2025-09-29T03:55:28.000Z

## Abstract

Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty and ambiguity. We propose MRAG-Suite, a diagnostic evaluation platform integrating diverse multimodal benchmarks (WebQA, Chart-RAG, Visual-RAG, MRAG-Bench). We introduce difficulty-based and ambiguity-aware filtering strategies, alongside MM-RAGChecker, a claim-level diagnostic tool. Our results demonstrate substantial accuracy reductions under difficult and ambiguous queries, highlighting prevalent hallucinations. MM-RAGChecker effectively diagnoses these issues, guiding future improvements in Visual RAG systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2509.24253>)

[PDF](<https://arxiv.org/pdf/2509.24253.pdf>)
