Spoken Multi-choice Question Answering on Spoken-MQA (test)
89Arithmetic Average AccuracyMPS-tbs
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| MPS-tbsmode=Think-Before-Speaking2025.10 | 89 | 90 | 88.4 | 94.4 | 93.2 | 93.6 | 93 | |
| MPS-thkfirstmode=Think-First2025.10 | 86.4 | 89 | 84.9 | 95.6 | 94.6 | 94.9 | 93.9 | |
| Whisper-Qwen2.5-Math-7B-Instruct2025.10 | 77.3 | — | — | — | — | 86.7 | 85.6 | |
| Mini-Omni-Reasoner2025.10 | 77.3 | 92.9 | 66.1 | 85.9 | 60.5 | 68.1 | 68.6 | |
| MPS-spkfirstmode=Speak-First2025.10 | 77.3 | 87 | 71.7 | 96 | 94.5 | 94.9 | 92.8 | |
| Whisper-Qwen2.5-7B-Instruct2025.10 | 70 | — | — | — | — | 72.5 | 72.2 | |
| Step-Audio 2reproducibility=reproduced by ourselves2025.10 | 65.9 | 89 | 52.6 | 95.6 | 90.4 | 91.9 | 88.8 | |
| Qwen2.5-Omni-7B2025.10 | 61.5 | 83 | 45.1 | 85.2 | 71.5 | 75.6 | 73.6 | |
| Qwen2.5-Omni-3B2025.10 | 60.1 | 84 | 43.3 | 81.5 | 57.1 | 64.4 | 63.6 | |
| MPS-wo/thkmode=without thinking process2025.10 | 47.6 | 71 | 34.1 | 88 | 67.8 | 73.8 | 70.6 | |
| Qwen2-Audio-7B-Instruct2025.10 | 36.3 | 43 | 31.2 | 55.4 | 22.5 | 32.3 | 32.7 | |
| GLM-4-Voice2025.10 | 30.1 | 40 | 22.5 | 54.4 | 28.5 | 36.2 | 35.3 | |
| Freeze-omni2025.10 | 26.8 | 43 | 14.5 | 69 | 19.8 | 34.4 | 33.3 | |
| LLaMA-Omni2025.10 | 23.5 | 40 | 11 | 29.5 | 10.5 | 16.2 | 16.8 | |
| Mini-Omni2025.10 | 3.5 | 5 | 2.3 | 0.8 | 1.9 | 1.6 | 1.7 |