Multiple-choice audio understanding on MMAU mini (test)
78.4Average AccuracySpectCount (Audio Flamingo 3)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SpectCount (Audio Flamingo 3)Size=8.3B, Trained Parameters=26.2M, Setting=SpectCount2026.06 | 78.4 | 83.18 | 77.54 | 74.47 | |
| Audio Flamingo 3Size=8.3B, Setting=Base (reproduced)2026.06 | 73.9 | 81.08 | 71.86 | 68.77 | |
| Audio Flamingo 3Size=8.3B, Setting=Base (reported)2026.06 | 73.3 | 79.58 | 73.95 | 66.37 | |
| Qwen2.5-OmniDecoding=TCD2026.04 | 73.2 | 79 | 71 | 69.7 | |
| Qwen2.5-OmniDecoding=Greedy2026.04 | 71.5 | 78.1 | 65.9 | 70.6 | |
| Qwen2.5-OmniDecoding=AAD, alpha=0.52026.04 | 71 | 78.1 | 68 | 67 | |
| Gemini 2.0 Flash2026.04 | 70.5 | 71.2 | 65.3 | 75.1 | |
| Qwen2.5-OmniDecoding=AAD, alpha=1.02026.04 | 70.4 | 75.1 | 68.6 | 67.6 | |
| Qwen2-Audio-Inst.Decoding=TCD2026.04 | 63.8 | 66.1 | 63 | 62.5 | |
| SpectCount (Qwen2-Audio-Instruct)Size=8.4B, Trained Parameters=25.9M, Setting=SpectCount2026.06 | 63.6 | 70.57 | 58.38 | 61.86 | |
| GPT-4o Audio2026.04 | 62.5 | 64.6 | 56.3 | 66.7 | |
| Qwen2-Audio-Inst.Decoding=Greedy2026.04 | 62.3 | 65.1 | 61.7 | 60 | |
| Qwen2-Audio-InstructSize=8.4B, Setting=Base (reported)2026.06 | 59.6 | 67.27 | 56.29 | 55.26 | |
| Qwen2-Audio-InstructSize=8.4B, Setting=Base (reproduced)2026.06 | 58.2 | 66.67 | 57.19 | 50.75 | |
| Global MergeKeep Ratio=0.752026.05 | 56.4 | — | — | — | |
| LTBM (local)Keep Ratio=0.752026.05 | 55.6 | — | — | — | |
| BaselineKeep Ratio=0.752026.05 | 55.4 | — | — | — | |
| BaselineKeep Ratio=0.52026.05 | 55.4 | — | — | — | |
| BaselineKeep Ratio=0.252026.05 | 55.4 | — | — | — | |
| GPT-4o mini Audio2026.04 | 53 | 50.8 | 39.2 | 69.1 | |
| Global MergeKeep Ratio=0.52026.05 | 53 | — | — | — | |
| LTBM (local)Keep Ratio=0.52026.05 | 52.3 | — | — | — | |
| Segmentwise Top-KKeep Ratio=0.752026.05 | 51 | — | — | — | |
| UniAvgKeep Ratio=0.52026.05 | 50.2 | — | — | — | |
| Global Top-KKeep Ratio=0.752026.05 | 49.3 | — | — | — | |
| Global MergeKeep Ratio=0.252026.05 | 47.7 | — | — | — | |
| LTBM (local)Keep Ratio=0.252026.05 | 47.2 | — | — | — | |
| Segmentwise Top-KKeep Ratio=0.52026.05 | 47.2 | — | — | — | |
| Global Top-KKeep Ratio=0.52026.05 | 46.6 | — | — | — | |
| Global Top-KKeep Ratio=0.252026.05 | 43.3 | — | — | — | |
| Mini-OmniDecoding=TCD2026.04 | 42.8 | 48.7 | 34.4 | 45.4 | |
| Segmentwise Top-KKeep Ratio=0.252026.05 | 42.7 | — | — | — | |
| Mini-OmniDecoding=Greedy2026.04 | 41.2 | 46.6 | 33.8 | 43.5 | |
| UniAvgKeep Ratio=0.252026.05 | 39.5 | — | — | — | |
| SALMONN2026.04 | 34.9 | 41.1 | 37.1 | 26.4 | |
| GAMA-IT2026.04 | 22.8 | 30.9 | 26.7 | 10.8 | |
| GAMA2026.04 | 20.8 | 31.8 | 17.7 | 12.9 | |
| LTU2026.04 | 17.4 | 20.4 | 16 | 15.9 | |
| Audio Flamingo Chat2026.04 | 16.6 | 25.2 | 17.7 | 6.9 |