Knowledge Evaluation on OpenBookQA (test)
92.31AccuracyQwen3-Omni-Instruct
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3-Omni-InstructType=Omni, Size=30B-A3B2026.02 | 92.31 | |
| Kimi-Audio-7B-InstructType=Audio, Size=7B2026.02 | 84.18 | |
| Qwen2.5-Omni-7BType=Omni, Size=7B2026.02 | 81.53 | |
| MiniCPM-oType=Omni, Size=9B2026.02 | 79.12 | |
| Full RepetitionAvg KV Cache=471.1, Avg Prefill FLOPs=3.140 T, Backbone=Qwen 2.5-3B2026.07 | 78 | |
| LLMLinguaAvg KV Cache=277.4, Avg Prefill FLOPs=2.105 T, Backbone=Qwen 2.5-3B, Repetition Type=Appending summary2026.07 | 77.8 | |
| PARTREPAvg KV Cache=280.4, Avg Prefill FLOPs=2.481 T, Backbone=Qwen 2.5-3B, τ=0.152026.07 | 77.8 | |
| Qwen2.5-Omni-3BType=Omni, Size=3B2026.02 | 77.36 | |
| No RepetitionAvg KV Cache=235.5, Avg Prefill FLOPs=1.549 T, Backbone=Qwen 2.5-3B2026.07 | 76.6 | |
| Naïve SummaryAvg KV Cache=277.4, Avg Prefill FLOPs=3.676 T, Backbone=Qwen 2.5-3B, Repetition Type=Appending summary2026.07 | 76.5 | |
| H2O EvictionAvg KV Cache=270.8, Avg Prefill FLOPs=3.162 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.07 | 76.2 | |
| Step-Audio-2-miniType=Audio, Size=8B2026.02 | 75.6 | |
| Ming-Lite-Omni-1.5Type=Omni, Size=19B-A2.8B2026.02 | 69.67 | |
| Echo EvictionAvg KV Cache=235.5, Avg Prefill FLOPs=3.152 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.07 | 69.6 | |
| Eureka-Audio-InstructType=Ours, Size=1.7B2026.02 | 69.23 | |
| Audio Flamingo 3Type=Audio, Size=8B2026.02 | 61.54 | |
| Eureka-Audio-BaseType=Ours, Size=1.7B2026.02 | 52.53 | |
| Qwen2-AudioType=Audio, Size=7B2026.02 | 49.67 | |
| LLMLingua Comp.Avg KV Cache=78.6, Avg Prefill FLOPs=1.130 T, Backbone=Qwen 2.5-3B, Repetition Type=Compressing full repetition2026.07 | 18.8 |