Audio Reasoning on MMAU-Pro
61.9Average ScoreAudioToolAgent
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AudioToolAgentSource=Closed Source2025.10 | 61.9 | 54.96 | 72.57 | 87.09 | 52 | 69.57 | 76.14 | 57.14 | 46.15 | 81.03 | 30 | 36.8 | 40.23 | |
| Gemini 3 ProSource=Closed Source2025.10 | 60.72 | 50.76 | 69.68 | 83.39 | 38 | 73.91 | 73.86 | 71.43 | 32.62 | 59.66 | 24.88 | 60.96 | 78.16 | |
| Gemini 2.5 FlashSource=Closed Source2025.10 | 59.2 | 51.9 | 64.9 | 73.4 | 42.8 | 58.7 | 61.3 | 42.8 | 36.3 | 71.7 | 21.2 | 67.5 | 95.1 | |
| OmniJigsawvariant=CMM2026.04 | 58.59 | — | — | — | — | — | — | — | — | — | — | — | — | |
| OmniJigsawvariant=SMS2026.04 | 58.46 | — | — | — | — | — | — | — | — | — | — | — | — | |
| OmniJigsawvariant=JMI2026.04 | 58.33 | — | — | — | — | — | — | — | — | — | — | — | — | |
| AudioJigsaw2026.04 | 57.67 | — | — | — | — | — | — | — | — | — | — | — | — | |
| AudioToolAgent-OpenSource=Open Source2025.10 | 57.5 | 46.5 | 65.7 | 72.7 | 36 | 56.5 | 55.7 | 57.1 | 42.8 | 66.2 | 27.7 | 62.2 | 58.6 | |
| Qwen3-Omni-30B2026.04 | 56.61 | — | — | — | — | — | — | — | — | — | — | — | — | |
| HumanOmniV22026.04 | 53.49 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o AudioSource=Closed Source2025.10 | 52.5 | 44.7 | 63.1 | 68.2 | 40.4 | 43.5 | 62.5 | 57.1 | 21.4 | 57.5 | 32.6 | 43.2 | 82.5 | |
| Omni-R12026.04 | 52.36 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-OmniSource=Open Source2025.10 | 52.2 | 47.6 | 61.5 | 57.4 | 40 | 53.2 | 60.2 | 28.5 | 41.2 | 60 | 24.3 | 52.3 | 61.3 | |
| Audio Flamingo 3Source=Open Source2025.10 | 51.7 | 55.9 | 61.7 | 58.8 | 40 | 41.3 | 47.7 | 57.1 | 26.8 | 58.6 | 26 | 44.2 | 33.3 | |
| Kimi-AudioSource=Open Source2025.10 | 46.6 | 46 | 57.6 | 52.2 | 46 | 54.3 | 48.9 | 42.8 | 43.7 | 50.6 | 17.2 | 34.5 | 42.3 | |
| DeSTA2.5-AudioSource=Open Source2025.10 | 40.6 | 35.7 | 48.2 | 49.9 | 22 | 36.9 | 35.2 | 28.6 | 28 | 51 | 19.8 | 36.4 | 46.5 | |
| Audio ReasonerSource=Open Source2025.10 | 39.5 | 34.2 | 50.1 | 44 | 26 | 36.9 | 43.2 | 28.6 | 20.3 | 43.4 | 22.6 | 38.6 | 43.4 | |
| Phi-4-multimodalSource=Open Source2025.10 | 38.7 | 25.7 | 47.8 | 47.6 | 30 | 39.1 | 30.1 | 28.6 | 39.7 | 42.7 | 11.4 | 42.5 | 65.4 |