Audio-based Reasoning on BigBench Audio
97.6AccuracyFun-Realtime-Audiochat
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Fun-Realtime-Audiochat2026.07 | 97.6 | — | — | |
| Step-Audio R1.1 Realtime2026.07 | 97.6 | — | — | |
| Gemini 3.1 Flash Live PreviewSetting=High2026.07 | 96.6 | — | — | |
| Grok Voice Agent2026.07 | 93.3 | — | — | |
| Gemini 2.5 Flash Native Audio Dialog Thinking2026.07 | 90.7 | — | — | |
| Audex 30B-A3B2026.07 | 90 | — | — | |
| Nova 2.0 SonicRelease=Mar 20262026.07 | 88.1 | — | — | |
| GPT Realtime2026.07 | 83.3 | — | — | |
| GPT-Realtime-1.52026.07 | 81.4 | — | — | |
| Serialthinking_mode=Think, variant=upper bound2026.01 | 73.77 | 76,000 | 84,000 | |
| Qwen3.5 Omni Plus Realtime2026.07 | 73 | — | — | |
| LTS-VoiceAgent2026.01 | 71.9 | 209 | 338 | |
| PredGenthinking_mode=Think2026.01 | 71.6 | 33,000 | 46,000 | |
| LTS-VADthinking_mode=Think2026.01 | 71.5 | 33,000 | 45,000 | |
| Gemini 3.1 Flash Live PreviewSetting=Minimal2026.07 | 71.3 | — | — | |
| GPT-4o mini RealtimeRelease=Dec 20242026.07 | 68.9 | — | — | |
| Gemini 2.5 Flash Native Audio Dialog2026.07 | 68.6 | — | — | |
| Audex 2B2026.07 | 64.3 | — | — | |
| GPT Realtime MiniRelease=Oct 20252026.07 | 63.6 | — | — | |
| MiMo-Audio2026.07 | 60.2 | — | — | |
| Qwen3.5 Omni Flash Realtime2026.07 | 59 | — | — | |
| Qwen3 Omni Flash2026.07 | 58.7 | — | — | |
| Qwen3 Omni Realtime2026.07 | 56.8 | — | — | |
| LTS-VADthinking_mode=No-think2026.01 | 55.2 | 206 | 258 | |
| Serialthinking_mode=No-think2026.01 | 54.95 | 990 | 3,166 | |
| PredGenthinking_mode=No-think2026.01 | 54.5 | 213 | 264 | |
| GPT-4o audio chatcompletions2026.07 | 54.3 | — | — | |
| Kimi-Audio-Instruct2026.07 | 51 | — | — | |
| Qwen2.5-Omniarchitecture=E2E2026.01 | 43.1 | 195.13 | 332.92 | |
| Nemotron VoiceChat2026.07 | 38.8 | — | — | |
| Freeze-Omni2026.07 | 33.4 | — | — | |
| PersonaPlex2026.07 | 19.1 | — | — | |
| FLM-Audio2026.07 | 16 | — | — | |
| Moshi2026.07 | 4.4 | — | — |