# Audio-Language Understanding (MCQ) on MMAU-Pro 1.0 (test) Source: [Audio-Language Understanding (MCQ) on MMAU-Pro 1.0 (test)]() Collection index: [llms.txt]() Data updated: 2026-04-28T10:21:24.887Z Evaluation of Audio-Language Understanding capabilities using Multiple Choice Questions (MCQ) on the test split of the MMAU-Pro 1.0 dataset, reporting Accuracy and Text-prior rate (RTP). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Audio-Language Understanding (MCQ) | | Dataset (including any stated split) | MMAU-Pro 1.0 (test) | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [Text-prior Rate (RTP)](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Qwen3-Omni (I) | 59.5 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen2.5-Omni-7B | 57.7 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen3-Omni (T) | 56.5 | Not recorded | 69f03f7a55a3c32af89fa542 | | Audio-Flamingo-3 | 52.7 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen2-Audio-7B | 44.8 | Not recorded | 69f03f7a55a3c32af89fa542 | | Audio-Flamingo-3 | 44.1 | Not recorded | 69f03f7a55a3c32af89fa542 | | Phi-4-Multimodal | 43.7 | Not recorded | 69f03f7a55a3c32af89fa542 | | DeSTA-2.5 | 43.5 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen3-Omni (I) | 43.2 | Not recorded | 69f03f7a55a3c32af89fa542 | | Voxtral-Mini-3B | 41.7 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen3-Omni (I) | 41 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen3-Omni (T) | 40.5 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen2.5-Omni-7B | 39.3 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen3-Omni (T) | 33.8 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen2-Audio-7B | 31.4 | Not recorded | 69f03f7a55a3c32af89fa542 | | DeSTA-2.5 | 31.3 | Not recorded | 69f03f7a55a3c32af89fa542 | | Audio-Flamingo-3 | 31.2 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen2.5-Omni-7B | 31.2 | Not recorded | 69f03f7a55a3c32af89fa542 | | Voxtral-Mini-3B | 30 | Not recorded | 69f03f7a55a3c32af89fa542 | | Phi-4-Multimodal | 29.9 | Not recorded | 69f03f7a55a3c32af89fa542 | | Phi-4-Multimodal | 28.6 | Not recorded | 69f03f7a55a3c32af89fa542 | | Qwen2-Audio-7B | 28.2 | Not recorded | 69f03f7a55a3c32af89fa542 | | Chance Level | 25.9 | Not recorded | 69f03f7a55a3c32af89fa542 | | Chance Level | 25.9 | Not recorded | 69f03f7a55a3c32af89fa542 | | Chance Level | 25.9 | Not recorded | 69f03f7a55a3c32af89fa542 | | DeSTA-2.5 | 20.3 | Not recorded | 69f03f7a55a3c32af89fa542 | | Voxtral-Mini-3B | 20 | Not recorded | 69f03f7a55a3c32af89fa542 | ### Qwen3-Omni (I) Accuracy: 59.5. Result ID: 69f089b247f34ce924d71f94. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"30B†","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Qwen2.5-Omni-7B Accuracy: 57.7. Result ID: 69f089b247f34ce924d71f91. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"10.7B","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Qwen3-Omni (T) Accuracy: 56.5. Result ID: 69f089b247f34ce924d71f97. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"30B†","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Audio-Flamingo-3 Accuracy: 52.7. Result ID: 69f089b247f34ce924d71f85. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.4B","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Qwen2-Audio-7B Accuracy: 44.8. Result ID: 69f089b247f34ce924d71f8e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.2B","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Audio-Flamingo-3 Accuracy: 44.1. Result ID: 69f089b247f34ce924d71f86. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.4B","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Phi-4-Multimodal Accuracy: 43.7. Result ID: 69f089b247f34ce924d71f8b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"5.6B","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### DeSTA-2.5 Accuracy: 43.5. Result ID: 69f089b247f34ce924d71f88. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.8B","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Qwen3-Omni (I) Accuracy: 43.2. Result ID: 69f089b247f34ce924d71f95. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"30B†","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Voxtral-Mini-3B Accuracy: 41.7. Result ID: 69f089b247f34ce924d71f9a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4.7B","Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Qwen3-Omni (I) Accuracy: 41. Result ID: 69f089b247f34ce924d71f96. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"30B†","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Qwen3-Omni (T) Accuracy: 40.5. Result ID: 69f089b247f34ce924d71f98. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"30B†","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Qwen2.5-Omni-7B Accuracy: 39.3. Result ID: 69f089b247f34ce924d71f92. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"10.7B","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Qwen3-Omni (T) Accuracy: 33.8. Result ID: 69f089b247f34ce924d71f99. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"30B†","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Qwen2-Audio-7B Accuracy: 31.4. Result ID: 69f089b247f34ce924d71f8f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.2B","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### DeSTA-2.5 Accuracy: 31.3. Result ID: 69f089b247f34ce924d71f89. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.8B","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Audio-Flamingo-3 Accuracy: 31.2. Result ID: 69f089b247f34ce924d71f87. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.4B","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Qwen2.5-Omni-7B Accuracy: 31.2. Result ID: 69f089b247f34ce924d71f93. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"10.7B","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Voxtral-Mini-3B Accuracy: 30. Result ID: 69f089b247f34ce924d71f9b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4.7B","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Phi-4-Multimodal Accuracy: 29.9. Result ID: 69f089b247f34ce924d71f8d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"5.6B","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Phi-4-Multimodal Accuracy: 28.6. Result ID: 69f089b247f34ce924d71f8c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"5.6B","Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Qwen2-Audio-7B Accuracy: 28.2. Result ID: 69f089b247f34ce924d71f90. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.2B","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Chance Level Accuracy: 25.9. Result ID: 69f089b247f34ce924d71f9d. [Reporting paper]() Recorded evaluation configuration: {"Size":null,"Evaluation Protocol":"Full"}. This record does not assert independent reproduction. ### Chance Level Accuracy: 25.9. Result ID: 69f089b247f34ce924d71f9e. [Reporting paper]() Recorded evaluation configuration: {"Size":null,"Evaluation Protocol":"None"}. This record does not assert independent reproduction. ### Chance Level Accuracy: 25.9. Result ID: 69f089b247f34ce924d71f9f. [Reporting paper]() Recorded evaluation configuration: {"Size":null,"Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### DeSTA-2.5 Accuracy: 20.3. Result ID: 69f089b247f34ce924d71f8a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8.8B","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. ### Voxtral-Mini-3B Accuracy: 20. Result ID: 69f089b247f34ce924d71f9c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4.7B","Evaluation Protocol":"TB"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.