Speech Understanding on MMSU
81.3AccuracyGemini-3.1-Pro
Evaluation results
| Method | Links | |
|---|---|---|
| Gemini-3.1-ProSize=–, Model Category=Closed-source models2026.06 | 81.3 | |
| Gemini-3-ProSize=–, Model Category=Closed-source models2026.06 | 81.28 | |
| MOSS-Audio-8B-ThinkingSize=8B, Model Category=Open-source models2026.06 | 75.52 | |
| Step-Audio-R1Size=33B, Model Category=Open-source models2026.06 | 75.18 | |
| MOSS-Audio-4B-ThinkingSize=4B, Model Category=Open-source models2026.06 | 71.2 | |
| Qwen3-Omni-30B-A3B-InstructSize=30B, Model Category=Open-source models2026.06 | 69 | |
| MOSS-Audio-8B-InstructSize=8B, Model Category=Open-source models2026.06 | 66.36 | |
| Step-Audio-R1.1Size=33B, Model Category=Open-source models2026.06 | 64.18 | |
| SpectCount (Audio Flamingo 3)Size=8.3B, Trained Parameters=26.2M, Setting=SpectCount2026.06 | 63.18 | |
| Kimi-AudioFull Duplex=false2026.07 | 62.2 | |
| MiMo-Audio-7BSize=7B, Model Category=Open-source models2026.06 | 61.94 | |
| Audio Flamingo 3Size=8.3B, Setting=Base (reproduced)2026.06 | 61.92 | |
| Audio Flamingo 3Size=8.3B, Setting=Base (reported)2026.06 | 61.4 | |
| Qwen2.5-Omni-3BModel Scale=3B2026.06 | 61.4 | |
| Qwen2.5-OmniSize=7B, Model Category=Open-source models2026.06 | 61.32 | |
| MiniCPM-o-4.5Size=9B, Model Category=Open-source models2026.06 | 60.96 | |
| Audio Flamingo 3Size=7B, Model Category=Open-source models2026.06 | 60.04 | |
| MOSS-Audio-4B-InstructSize=4B, Model Category=Open-source models2026.06 | 59.68 | |
| GPT4o-AudioSize=–, Model Category=Closed-source models2026.06 | 58.76 | |
| Audio Flamingo NextSize=8B, Model Category=Open-source models2026.06 | 57.2 | |
| DuplexPOFull Duplex=true2026.07 | 56.2 | |
| SFT BaselineFull Duplex=true2026.07 | 54.9 | |
| Kimi-AudioSize=7B, Model Category=Open-source models2026.06 | 54.74 | |
| SpectCount (Qwen2-Audio-Instruct)Size=8.4B, Trained Parameters=25.9M, Setting=SpectCount2026.06 | 54.24 | |
| Qwen2-Audio-InstructSize=8.4B, Setting=Base (reported)2026.06 | 53.27 | |
| Baichuan-AudioFull Duplex=false2026.07 | 53.2 | |
| Gemma-4-E2B-itModel Scale=4B2026.06 | 52.8 | |
| AVLM-2BModel Scale=2B2026.06 | 51.9 | |
| Qwen2-Audio-InstructSize=8.4B, Setting=Base (reproduced)2026.06 | 48.44 | |
| GLM-4-VoiceFull Duplex=false2026.07 | 39.8 | |
| Qwen2-AudioFull Duplex=false2026.07 | 35.7 | |
| SALMONN-omniFull Duplex=true2026.07 | 30 | |
| Freeze-OmniFull Duplex=true2026.07 | 28.1 | |
| SALM-DuplexFull Duplex=true2026.07 | 26.3 | |
| MoshiFull Duplex=true2026.07 | 24 |