# MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems Source: [MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems]() Collection index: [llms.txt]() Data updated: 2026-05-21T05:00:05.413Z Authors: Quanxing Xu, Yuhao Tian, Ling Zhou, Xian Zhong, Xiaohua Huang, Rubing Huang, Chia-Wen Lin Published: 2026-05-19T03:37:54.000Z ## Abstract Visual Question Answering (VQA), as the representative multimodal task, serves as a key benchmark for evaluating the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, existing evaluations largely rely on static datasets and accuracy-based metrics, which fail to capture robustness, consistency, and generalization. Inspired by Metamorphic Testing (MT), we propose Metamorphic Robustness Assessment (MetaRA), a testing framework that employs Metamorphic Relations (MRs) to systematically probe vulnerabilities in MLLM-based VQA systems. MetaRA generates controlled variations of image-question inputs based on specific MRs and evaluates models across diverse conditions. Applying MetaRA to multiple MLLM-based VQA models across different tasks reveals nuanced failure patterns, including sensitivity to linguistic perturbations, over-reliance on superficial visual cues, and deeper weaknesses in multimodal reasoning. Experimental results demonstrate that MetaRA provides richer diagnostic insights than conventional accuracy metrics, exposing failure modes that remain hidden under standard benchmarks. Overall, this work highlights the need for systematic robustness evaluation in VQA and positions metamorphic assessment as a scalable, model-agnostic approach toward trustworthy multimodal AI. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Knowledge-Based Visual Question Answering on E-VQA](): Knowledge-based Visual Question Answering; E-VQA; Final Fidelity Rate: not recorded. - [Knowledge-Based Visual Question Answering on INFOSEEK](): Knowledge-based Visual Question Answering; InfoSeek; FR (All): not recorded. - [OCR-VQA on DOCVQA](): OCR VQA; DocVQA; FR: not recorded. - [OCR-VQA on INFOVQA](): OCR VQA; InfoVQA; FR: not recorded. - [OCR-VQA on CHARTQA](): OCR VQA; ChartQA; FR: not recorded. - [OCR-VQA on TEXTVQA](): OCR VQA; TextVQA; FR: not recorded. - [OCR-VQA Average on DOCVQA, INFOVQA, CHARTQA, and TEXTVQA](): OCR VQA; DOCVQA, INFOVQA, CHARTQA, and TEXTVQA Average; Final FR: not recorded. ## Original sources and implementation [Original paper]() [PDF]()