Audio-Language Understanding (MCQ) on MMAU-Pro 1.0 (test)
59.5AccuracyQwen3-Omni (I)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3-Omni (I)Size=30B†, Evaluation Protocol=Full2026.04 | 59.5 | 72.6 | |
| Qwen2.5-Omni-7BSize=10.7B, Evaluation Protocol=Full2026.04 | 57.7 | 68.2 | |
| Qwen3-Omni (T)Size=30B†, Evaluation Protocol=Full2026.04 | 56.5 | 71.7 | |
| Audio-Flamingo-3Size=8.4B, Evaluation Protocol=Full2026.04 | 52.7 | 83.7 | |
| Qwen2-Audio-7BSize=8.2B, Evaluation Protocol=Full2026.04 | 44.8 | 70.1 | |
| Audio-Flamingo-3Size=8.4B, Evaluation Protocol=None2026.04 | 44.1 | — | |
| Phi-4-MultimodalSize=5.6B, Evaluation Protocol=Full2026.04 | 43.7 | 65.5 | |
| DeSTA-2.5Size=8.8B, Evaluation Protocol=Full2026.04 | 43.5 | 72 | |
| Qwen3-Omni (I)Size=30B†, Evaluation Protocol=None2026.04 | 43.2 | — | |
| Voxtral-Mini-3BSize=4.7B, Evaluation Protocol=Full2026.04 | 41.7 | 71.9 | |
| Qwen3-Omni (I)Size=30B†, Evaluation Protocol=TB2026.04 | 41 | — | |
| Qwen3-Omni (T)Size=30B†, Evaluation Protocol=None2026.04 | 40.5 | — | |
| Qwen2.5-Omni-7BSize=10.7B, Evaluation Protocol=None2026.04 | 39.3 | — | |
| Qwen3-Omni (T)Size=30B†, Evaluation Protocol=TB2026.04 | 33.8 | — | |
| Qwen2-Audio-7BSize=8.2B, Evaluation Protocol=None2026.04 | 31.4 | — | |
| DeSTA-2.5Size=8.8B, Evaluation Protocol=None2026.04 | 31.3 | — | |
| Audio-Flamingo-3Size=8.4B, Evaluation Protocol=TB2026.04 | 31.2 | — | |
| Qwen2.5-Omni-7BSize=10.7B, Evaluation Protocol=TB2026.04 | 31.2 | — | |
| Voxtral-Mini-3BSize=4.7B, Evaluation Protocol=None2026.04 | 30 | — | |
| Phi-4-MultimodalSize=5.6B, Evaluation Protocol=TB2026.04 | 29.9 | — | |
| Phi-4-MultimodalSize=5.6B, Evaluation Protocol=None2026.04 | 28.6 | — | |
| Qwen2-Audio-7BSize=8.2B, Evaluation Protocol=TB2026.04 | 28.2 | — | |
| Chance LevelEvaluation Protocol=Full2026.04 | 25.9 | — | |
| Chance LevelEvaluation Protocol=None2026.04 | 25.9 | — | |
| Chance LevelEvaluation Protocol=TB2026.04 | 25.9 | — | |
| DeSTA-2.5Size=8.8B, Evaluation Protocol=TB2026.04 | 20.3 | — | |
| Voxtral-Mini-3BSize=4.7B, Evaluation Protocol=TB2026.04 | 20 | — |