# Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory Source: [Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory]() Collection index: [llms.txt]() Data updated: 2026-03-05T05:00:04.307Z Authors: Shunki Uebayashi, Kento Masui, Kyohei Atarashi, Han Bao, Hisashi Kashima, Naoto Inoue, Mayu Otani, Koh Takeuchi Published: 2026-03-03T06:51:08.000Z ## Abstract Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Model Ranking Correlation Analysis on MMMU (10%-50% subsets)](): Model Ranking Correlation Analysis; MMMU; Spearman Correlation (10% Subset): 0.96. - [Model Ranking Correlation Analysis on MathVista](): Model Ranking Correlation Analysis; MathVista; Spearman Correlation (10% subset): 0.92. - [Model Ranking Correlation Analysis on SEED-Bench (10%-50% subsets)](): Model Ranking Correlation Analysis; SEED-Bench; Spearman Correlation (10% subset): 0.94. ## Original sources and implementation [Original paper]() [PDF]()