Language Understanding on MMLU-Pro (Subject Scores Breakdown)
18.5Math Score (MMLU-Pro)POLAR
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| POLARBackbone=Qwen1.5-MoE-A2.7B-Chat, Decoding protocol=pass@12026.06 | 18.5 | 20.3 | 18.3 | 20.4 | 19.9 | 26.6 | 34.6 | 29.5 | 35.3 | 20.9 | 36.9 | 25.9 | 23.5 | |
| DR.LLMBackbone=Qwen1.5-MoE-A2.7B-Chat, Decoding protocol=pass@12026.06 | 14.6 | 17.4 | 13.2 | 19.8 | 16 | 20.8 | 31.8 | 27 | 32.2 | 17.5 | 33.5 | 22.8 | 21.3 | |
| BaseBackbone=Qwen1.5-MoE-A2.7B-Chat, Decoding protocol=pass@1, Temperature (τ)=02026.06 | 13.9 | 15.6 | 13.8 | 16.6 | 15.1 | 22.8 | 31 | 26.8 | 30.7 | 18.4 | 34.7 | 22.8 | 22.6 | |
| ShortGPTBackbone=Qwen1.5-MoE-A2.7B-Chat, Decoding protocol=pass@12026.06 | 3.8 | 0.4 | 4.2 | 3.3 | 3.8 | 2.6 | 2.4 | 2.2 | 2.8 | 4.9 | 2.9 | 1.8 | 4.7 | |
| FlexiDepthBackbone=Qwen1.5-MoE-A2.7B-Chat, Decoding protocol=pass@12026.06 | 2.1 | 3.5 | 2.5 | 4.4 | 3.3 | 2.5 | 1.7 | 3.5 | 2 | 2.9 | 3.8 | 3 | 3.7 | |
| MindSkipBackbone=Qwen1.5-MoE-A2.7B-Chat, Decoding protocol=pass@12026.06 | 0.9 | 1.2 | 1 | 1.9 | 1.3 | 1.3 | 0.6 | 0.7 | 1.3 | 0.8 | 1.7 | 0.6 | 1.8 |