Speech Reasoning and Response Generation on M.J. Score Speech Understanding Benchmark
8.12Overall M.J. ScoreGemini 2.5 Pro
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gemini 2.5 ProPrompt Style=CoT2025.12 | 8.12 | 8.02 | 8.28 | 82.47 | 51.29 | 112.62 | |
| SWM (Llama3.1-8b)Prompt Style=CoT, Backbone=Llama3.1-8b2025.12 | 7.81 | 7.84 | 7.76 | 97.8 | 66.26 | 105.7 | |
| SWM (Qwen2-Audio)Prompt Style=CoT, Backbone=Qwen2-Audio2025.12 | 7.59 | 7.26 | 8.08 | 91.8 | 71.02 | 104.64 | |
| GPT-4oPrompt Style=CoT2025.12 | 7.41 | 6.98 | 8.06 | 68.2 | 45.16 | 105.23 | |
| Qwen2-Audio-CoTPrompt Style=CoT, Note=Tuned Baseline2025.12 | 5.18 | 4.76 | 5.82 | 92.11 | 34.72 | 102.44 | |
| VoxtralPrompt Style=CoT2025.12 | 2.92 | 2.52 | 3.52 | 10.89 | 5.56 | 71.42 | |
| VoxtralPrompt Style=Direct2025.12 | 2.89 | 2.46 | 3.54 | 10.28 | 5.88 | 69.64 | |
| Qwen-AudioPrompt Style=Direct2025.12 | 2.7 | 2.2 | 3.46 | 14.2 | 8 | 28.99 | |
| Qwen2-AudioPrompt Style=Direct2025.12 | 2.63 | 2.08 | 3.47 | 5.14 | 15.38 | 15.6 | |
| Qwen2-AudioPrompt Style=CoT2025.12 | 2.39 | 1.96 | 3.04 | 6.11 | 17.5 | 21.19 |