Audio Question Answering on MMAR
80.4Average ScoreAUDIO-MIND
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| AUDIO-MINDEvaluation Protocol=evaluated by authors (1,000 examples)2026.05 | 80.4 | 78.18 | 63.11 | 87.41 | 100 | 87.61 | 79.27 | 87.5 | |
| Qwen3.5-OmniEvaluation Protocol=original2026.05 | 80 | — | — | — | — | — | — | — | |
| Qwen3.5-OmniEvaluation Protocol=evaluated by authors (1,000 examples)2026.05 | 78.9 | 75.76 | 61.17 | 85.37 | 90.91 | 87.61 | 79.27 | 87.5 | |
| AudioToolAgentEvaluation Protocol=original2026.05 | 77 | 73.33 | 63.59 | 82.99 | 63.64 | 83.03 | 81.71 | 79.17 | |
| Gemini 3 ProEvaluation Protocol=original2026.05 | 76.4 | 72.73 | 61.65 | 84.35 | 72.73 | 81.19 | 78.05 | 83.33 | |
| Gemini 2.5 ProEvaluation Protocol=original2026.05 | 74.7 | 67.3 | 56.8 | 82 | 100 | 84.9 | 80.5 | 66.7 | |
| AudioGenie-ReasonerBackbone=Matched Backbone, Evaluation Protocol=evaluated by authors (1,000 examples)2026.05 | 70.5 | 70.91 | 54.85 | 79.59 | 54.55 | 72.48 | 76.83 | 58.33 | |
| Gemini-2.5-Pro2026.02 | 70.03 | 66.06 | 57.77 | 73.13 | 81.82 | 71.1 | 69.51 | 70.83 | |
| EchoSize=7B, Reasoning strategy=audio-interleaved2026.02 | 69.99 | 67.27 | 60.68 | 69.39 | 81.82 | 69.72 | 74.39 | 66.67 | |
| Gemini-2.0-Flash2026.02 | 67.9 | 61.21 | 50.97 | 72.11 | 81.82 | 72.48 | 65.85 | 70.83 | |
| Audio-ThinkerSize=7B2026.02 | 67.25 | 68.48 | 53.88 | 64.29 | 72.73 | 71.56 | 73.17 | 66.67 | |
| Gemini 2.0 FlashEvaluation Protocol=original2026.05 | 65.6 | 61.21 | 50.97 | 72.11 | 81.82 | 72.48 | 65.85 | 70.83 | |
| AudioRouterEvaluation Protocol=original2026.05 | 64.4 | 61.4 | 66.1 | 77.2 | 57.2 | 65.1 | 68.3 | 55.5 | |
| Gemini 2.5 ProModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 64.1 | — | — | — | — | — | — | — | |
| Gemini 2.5 ProModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 64.1 | — | — | — | — | — | — | — | |
| GPT-4o-Audio2026.02 | 64.09 | 53.94 | 50.97 | 70.41 | 63.64 | 72.48 | 62.2 | 75 | |
| GPT-4o AudioEvaluation Protocol=original2026.05 | 63.5 | 53.94 | 50.97 | 70.41 | 63.64 | 72.48 | 62.2 | 75 | |
| Omni-R1Size=7B2026.02 | 63.46 | 67.3 | 51.5 | 64.3 | 55.5 | 70.2 | 64.6 | 70.8 | |
| AQA-TTRLBackbone=Qwen2.5-Omni 7B, Protocol=Self-Adaptation2025.10 | 63.2 | — | — | — | — | — | — | — | |
| AuTAgentEvaluation Protocol=original2026.05 | 63 | 51.52 | 45.15 | 73.81 | — | — | — | — | |
| Audio-Captioner-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 59.8 | — | — | — | — | — | — | — | |
| Audio-Captioner-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 59.8 | — | — | — | — | — | — | — | |
| GPT-4o AudioModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 59.3 | — | — | — | — | — | — | — | |
| GPT-4o AudioModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 59.3 | — | — | — | — | — | — | — | |
| DIMVBackbone=Qwen2.5-Omni 7B, Protocol=Direct Inference with Majority Voting2025.10 | 58.9 | — | — | — | — | — | — | — | |
| AudioGenie-ReasonerEvaluation Protocol=original2026.05 | 58.85 | 49.68 | 43.26 | 69.23 | 45.45 | 64.53 | 65.33 | 59.09 | |
| SFTBackbone=Qwen2.5-Omni 7B, Protocol=Supervised Fine-Tuning2025.10 | 58.6 | — | — | — | — | — | — | — | |
| Gemini 2.5 FlashModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 58.2 | — | — | — | — | — | — | — | |
| Gemini 2.5 FlashModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 58.2 | — | — | — | — | — | — | — | |
| AQA-TTRLBackbone=Qwen2.5-Omni 3B, Protocol=Self-Adaptation2025.10 | 57.9 | — | — | — | — | — | — | — | |
| DIBackbone=Qwen2.5-Omni 7B, Protocol=Direct Inference2025.10 | 57.7 | — | — | — | — | — | — | — | |
| Qwen2.5-OmniSize=7B2026.02 | 57.33 | 58.79 | 40.78 | 59.86 | 54.55 | 61.93 | 67.07 | 58.33 | |
| Qwen2.5-Omni-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 51.8 | — | — | — | — | — | — | — | |
| Qwen2.5-Omni-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 51.8 | — | — | — | — | — | — | — | |
| MiDashengLM-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 50.7 | — | — | — | — | — | — | — | |
| MiDashengLM-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 50.7 | — | — | — | — | — | — | — | |
| Gemini 2.0 FlashModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 50.6 | — | — | — | — | — | — | — | |
| Gemini 2.0 FlashModel Category=Proprietary Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 50.6 | — | — | — | — | — | — | — | |
| GPT-4o-mini-Audio2026.02 | 49.49 | 38.79 | 35.92 | 58.84 | 45.45 | 60.09 | 57.32 | 50 | |
| SFTBackbone=Qwen2.5-Omni 3B, Protocol=Supervised Fine-Tuning2025.10 | 48.2 | — | — | — | — | — | — | — | |
| DIBackbone=Qwen2.5-Omni 3B, Protocol=Direct Inference2025.10 | 46.9 | — | — | — | — | — | — | — | |
| DIMVBackbone=Qwen2.5-Omni 3B, Protocol=Direct Inference with Majority Voting2025.10 | 46.7 | — | — | — | — | — | — | — | |
| Qwen2-Audio-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 44.2 | — | — | — | — | — | — | — | |
| Qwen2-Audio-7BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 44.2 | — | — | — | — | — | — | — | |
| SALMONN-13BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 42.5 | — | — | — | — | — | — | — | |
| SALMONN-13BModel Category=Open-Source Models, QA Backbone=GPT-4o, Evaluation Protocol=caption-to-QA cascade2025.10 | 42.5 | — | — | — | — | — | — | — | |
| Baichuan-Omni-1.5Size=11B2026.02 | 40.05 | 41.21 | 33.01 | 40.48 | 36.36 | 48.62 | 39.02 | 41.67 | |
| Audio-ReasonerSize=8.4B2026.02 | 36.42 | 43.64 | 33.5 | 32.99 | 45.45 | 42.66 | 31.71 | 25 | |
| SALMONNSize=13B2026.02 | 31.01 | 30.3 | 31.07 | 34.69 | 9.09 | 34.86 | 35.37 | 41.67 | |
| Audio-CoTSize=8.4B2026.02 | 28.97 | 35.76 | 25.24 | 34.01 | 9.09 | 30.73 | 30.49 | 37.5 | |
| Random Guess2026.02 | 28.61 | 29.39 | 25.88 | 31.48 | 25 | 29.3 | 31.1 | 28.13 | |
| Qwen2-AudioSize=8.4B2026.02 | 26.53 | 33.33 | 24.27 | 32.31 | 9.09 | 31.19 | 30.49 | 25 | |
| GAMASize=7B2026.02 | 24.83 | 29.09 | 24.27 | 27.89 | 27.27 | 24.77 | 28.05 | 12.5 | |
| Qwen-AudioSize=8.4B2026.02 | 21.59 | 27.88 | 20.39 | 22.11 | 9.09 | 25.23 | 25.61 | 20.83 | |
| Audio-Flamingo-2Size=3B2026.02 | 19.91 | 24.85 | 17.48 | 20.75 | 18.18 | 26.61 | 23.17 | 8.33 |