Speech Understanding on FMSU-Bench
74Average ScoreGemini 3.1 Pro
Evaluation results
| Method | Links | |||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 3.1 ProModel Category=Proprietary Models2026.05 | 74 | 86.3 | 85.3 | 62.4 | 65.2 | 81.9 | 75.8 | 66.6 | 74.6 | 69.8 | 71.7 | 87.3 | 91.9 | 67.2 | 76.4 | 76 | 68.9 | 65.6 | 55.6 | 93.1 | 94.5 | 66.7 | 61.4 | 75.6 | 85.1 | 72.1 | 61.9 | 71 | 60.8 | |
| FM-SpeechModel Category=Open-source Models2026.05 | 72.8 | 99.1 | 99.3 | 75.2 | 79.2 | 83.5 | 75.1 | 55.2 | 78.4 | 52.9 | 72.2 | 74.4 | 78.6 | 63.7 | 63.6 | 62.5 | 63.3 | 60.4 | 65.4 | 79.9 | 72.5 | 65.8 | 55.8 | 70.8 | 77.1 | 87.5 | 69.9 | 77 | 79.8 | |
| Gemini 3 FlashModel Category=Proprietary Models2026.05 | 71.9 | 85.4 | 88.3 | 63.3 | 59.1 | 65.1 | 67 | 69.3 | 77.7 | 74.7 | 75.5 | 87.2 | 91.6 | 71.9 | 80.2 | 73.2 | 61.3 | 62.9 | 56.6 | 86.9 | 87.1 | 60.2 | 55.9 | 76.6 | 86.7 | 63.3 | 55.9 | 69.5 | 61.4 | |
| Qwen3-OmniModel Category=Open-source Models2026.05 | 69.4 | 99 | 99.5 | 70.4 | 64.8 | 67 | 54.3 | 68.2 | 73 | 75.6 | 74.2 | 76 | 84.4 | 58 | 70.4 | 70.6 | 59.4 | 57.3 | 57.6 | 80.1 | 80.2 | 57.6 | 61 | 72.2 | 72.4 | 65.2 | 59.5 | 61.1 | 53.4 | |
| Gemini 2.5 FlashModel Category=Proprietary Models2026.05 | 69 | 97 | 93.6 | 53.4 | 66.1 | 55.4 | 63 | 62.1 | 80.9 | 68.1 | 67 | 80.3 | 90.1 | 75.8 | 81.4 | 69.6 | 64.9 | 77.8 | 58.2 | 85.5 | 84.9 | 46.4 | 51.1 | 70.2 | 66.7 | 53.9 | 52.1 | 63.5 | 52.4 | |
| Omni-CaptionerModel Category=Open-source Models2026.05 | 66 | 98.4 | 98.8 | 54.6 | 48.6 | 54.3 | 65 | 65.4 | 71.1 | 58.8 | 74.5 | 81.5 | 84.3 | 60.2 | 59.5 | 70.1 | 75.7 | 61.9 | 58 | 79.9 | 79.7 | 47.8 | 38.8 | 54 | 70.6 | 58.2 | 45.5 | — | — | |
| Mimo-AudioModel Category=Open-source Models2026.05 | 64.1 | 88.4 | 96.9 | 59.9 | 57.1 | 63.5 | 64.8 | 59.8 | 64.6 | 58.6 | 65.9 | 78.3 | 81.5 | 53 | 60.2 | 70.4 | 64.3 | 58.5 | 54.6 | 76.1 | 76.6 | 46.5 | 50.3 | 75.8 | 79.2 | 50.6 | 41.7 | 50.8 | 47.4 | |
| Qwen2.5-OmniModel Category=Open-source Models2026.05 | 59.7 | 96.9 | 99.3 | 61 | 62.4 | 70.9 | 47.9 | 63.8 | 45.5 | 75.4 | 70.6 | 60.7 | 70.5 | 25.6 | 39 | 60.8 | 48.7 | 45.3 | 40.7 | 79.5 | 87 | 29.1 | 48.7 | 67.5 | 66.7 | 64.7 | 49.7 | 48.4 | 45 | |
| Kimi-AudioModel Category=Open-source Models2026.05 | 54.3 | 79.4 | 95.6 | 42.4 | 50.1 | 59.3 | 54.4 | 37.2 | 49.7 | 37.9 | 51.9 | 49.9 | 74.5 | 23.9 | 38.9 | 49.7 | 51.9 | 45.1 | 42.6 | 56 | 69.6 | 50.1 | 58.5 | 71 | 67.9 | 59.9 | 53 | 56.8 | 44.2 | |
| Step-Audio 2Model Category=Open-source Models2026.05 | 48.7 | 90.2 | 87.5 | 42.4 | 38.7 | 67.2 | 48 | 24.3 | 21.4 | 34.7 | 38.8 | 58.6 | 65.8 | 25.8 | 33.8 | 56.2 | 40.3 | 46 | 41.7 | 69.4 | 73 | 31.7 | 36.8 | 63.5 | 58.5 | 41.8 | 36.9 | 47.8 | 43.4 | |
| Audio Flamingo 3Model Category=Open-source Models2026.05 | 47.6 | 89.8 | 97.6 | 41.6 | 42.1 | 35.4 | 36.4 | 20.2 | 32.7 | 21.4 | 59 | 40.6 | 70 | 26.8 | 53.6 | 42.8 | 41.1 | 35.7 | 36.2 | 65.8 | 79.3 | 42.1 | 48.5 | 55.2 | 53.6 | 45.3 | 47.2 | 34.8 | 38 | |
| Qwen2-AudioModel Category=Open-source Models2026.05 | 23.6 | 46.9 | 36.8 | 15.3 | 14.6 | 21 | 19.9 | 15.8 | 14.6 | 22.3 | 21 | 28.7 | 17.1 | 26.1 | 25.8 | 20.1 | 14.4 | 16.9 | 13.9 | 44.3 | 38.9 | 12.2 | 20.9 | 34.5 | 21.6 | 14.5 | 26.1 | 30.7 | 25.5 |