Multimodal Audio Understanding on MMAU mini v05.15.25 (test)
79.6Sound AccuracyAudio Flamingo 3
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Audio Flamingo 3Size=8B, No LLM Modification=×, # Samples=26.7M, # Hours=54.4K2026.03 | 79.6 | 74 | 66.4 | 73.3 | |
| Qwen2.5-OmniSize=7B, No LLM Modification=×, # Tokens=0.3T2026.03 | 78.1 | 65.9 | 70.6 | 71.5 | |
| Audio Flamingo 2Size=3B, No LLM Modification=×, # Samples=5.9M2026.03 | 71.5 | 71 | 44.7 | 62.4 | |
| DeSTA-2.5-AudioSize=8B, No LLM Modification=✓, # Samples=5M, # Hours=7K2026.03 | 70.3 | 56.3 | 71.5 | 66 | |
| Audio ReasonerSize=8B, No LLM Modification=×, # Samples=> 1.2M, # Hours=> 320K2026.03 | 67.9 | 69.2 | 66.1 | 67.7 | |
| Qwen2-Audio-InstructSize=7B, No LLM Modification=×, # Hours=320K2026.03 | 67.3 | 56.3 | 55.3 | 59.6 | |
| ALARM-CASize=4B, No LLM Modification=✓, # Samples=5.5M, # Hours=17K, # Tokens=1.5B2026.03 | 66.4 | 57.2 | 60.1 | 61.2 | |
| Phi-4-MultimodalSize=4B, No LLM Modification=×, # Samples=> 28M, # Hours=> 2M2026.03 | 65.5 | 64.37 | 67.27 | 66.7 | |
| GPT-4o Audio2026.03 | 64.6 | 56.3 | 66.7 | 62.5 | |
| ALARM-ESize=4B, No LLM Modification=✓, # Samples=5.5M, # Hours=17K, # Tokens=1.5B2026.03 | 64 | 54.8 | 77.2 | 65.3 | |
| AL-SSLAM-RSize=4B, No LLM Modification=✓, # Samples=0.4M, # Hours=1.2K, # Tokens=108M2026.03 | 61.3 | 46.7 | 45.9 | 51.3 | |
| ALARM-PSize=4B, No LLM Modification=✓, # Samples=5.5M, # Hours=17K, # Tokens=1.5B2026.03 | 58.6 | 52.7 | 68.8 | 60 | |
| Gemma 3nSize=8B4E2026.03 | 55.9 | 56.9 | 61.3 | 58 | |
| AL-Whisper-RSize=4B, No LLM Modification=✓, # Samples=5.5M, # Hours=17K, # Tokens=1.5B2026.03 | 52.3 | 46.7 | 57.1 | 52 | |
| Gemma 3nSize=6B2E2026.03 | 51.4 | 52.1 | 52.2 | 51.7 | |
| GPT-4o mini Audio2026.03 | 50.8 | 39.2 | 69.1 | 53 | |
| AL-MuQ-RSize=4B, No LLM Modification=✓, # Samples=0.4M, # Hours=1.5K, # Tokens=137M2026.03 | 50.5 | 53.9 | 40.84 | 48.4 | |
| AL-W2VBERT2-RSize=4B, No LLM Modification=✓, # Samples=0.5M, # Hours=0.6K, # Tokens=58M2026.03 | 46.2 | 47.9 | 47.7 | 47.3 | |
| AL-Whisper-Instruct-RSize=4B, No LLM Modification=✓, # Samples=0.5M, # Hours=1K, # Tokens=91M2026.03 | 45.9 | 46.7 | 73 | 55.2 | |
| MusiLingoSize=7B, No LLM Modification=✓, # Samples=0.7M2026.03 | 43.2 | 40.1 | 31.2 | 38.2 | |
| SALMONNSize=13B, No LLM Modification=×, # Samples=2.3M, # Hours=4.4K2026.03 | 41.1 | 37.1 | 26.4 | 34.9 | |
| MU-LLaMASize=7B, No LLM Modification=✓, # Samples=0.1M, # Hours=762026.03 | 33 | 32.3 | 17.4 | 27.6 | |
| GAMASize=7B, No LLM Modification=×, # Samples=6.3M2026.03 | 31.8 | 17.7 | 12.9 | 20.8 | |
| GAMA-ITSize=7B, No LLM Modification=×, # Samples=6.5M2026.03 | 30.9 | 26.7 | 10.8 | 22.8 | |
| LTUSize=7B, No LLM Modification=×, # Samples=5.6M2026.03 | 20.4 | 16 | 15.9 | 17.4 |