Audio QA Correctness Assessment on MMAU and MMAR unseen question-based (test)
0.9103Spearman ρORCA (Gemma3-12B)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ORCA (Gemma3-12B)Backbone=Gemma3, Model Size=12B, Clamped=true2025.11 | 0.9103 | 0.8085 | 0.084 | 0.0199 | |
| Gemini-2.5-FlashModel Type=LLM Judge2025.11 | 0.8998 | 0.807 | 0.0911 | — | |
| Judge Fusion (-Gemini)Composition=Fusion excluding Gemini2025.11 | 0.8993 | 0.7682 | 0.1079 | — | |
| ORCA (OLMo2-7B)Backbone=OLMo2, Model Size=7B, Clamped=true2025.11 | 0.8992 | 0.79 | 0.0827 | 0.0179 | |
| ORCA (Gemma3-4B)Backbone=Gemma3, Model Size=4B, Clamped=true2025.11 | 0.8981 | 0.7853 | 0.0945 | 0.0218 | |
| ORCA (OLMo2-7B)Backbone=OLMo2, Model Size=7B, Clamped=false2025.11 | 0.8973 | 0.7631 | 0.0861 | 0.0179 | |
| Average judge (+Gemini)Composition=Average including Gemini2025.11 | 0.8972 | 0.7704 | 0.1078 | 0.0283 | |
| ORCA (OLMo2-1B)Backbone=OLMo2, Model Size=1B, Clamped=true2025.11 | 0.8953 | 0.7842 | 0.0943 | 0.0223 | |
| Gemma3-12BModel Type=LLM Judge, Model Size=12B2025.11 | 0.8927 | 0.7989 | 0.1109 | — | |
| ORCA (Llama3.2-1B)Backbone=Llama3.2, Model Size=1B, Clamped=true2025.11 | 0.8926 | 0.7809 | 0.0933 | 0.0226 | |
| Average judge (-Gemini)Composition=Average excluding Gemini2025.11 | 0.8902 | 0.7614 | 0.1172 | 0.0296 | |
| ORCA (Gemma3-1B)Backbone=Gemma3, Model Size=1B, Clamped=true2025.11 | 0.889 | 0.7704 | 0.0967 | 0.0197 | |
| ORCA (OLMo2-1B)Backbone=OLMo2, Model Size=1B, Clamped=false2025.11 | 0.8877 | 0.7418 | 0.1004 | 0.0223 | |
| Qwen2.5-7BModel Type=LLM Judge, Model Size=7B2025.11 | 0.8553 | 0.7514 | 0.1205 | — | |
| ORCA (Gemma3-270M)Backbone=Gemma3, Model Size=270M, Clamped=true2025.11 | 0.8444 | 0.7209 | 0.121 | 0.0229 | |
| Llama3.1-8BModel Type=LLM Judge, Model Size=8B2025.11 | 0.8435 | 0.7325 | 0.1279 | — | |
| Gemma3-4BModel Type=LLM Judge, Model Size=4B2025.11 | 0.8149 | 0.7095 | 0.1471 | — | |
| Prometheus2-7BModel Type=LLM Judge, Model Size=7B2025.11 | 0.7439 | 0.6362 | 0.1941 | — |