Multilingual Language Understanding on MMMLU
76.1CLCallBase
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| BaseBackbone=Qwen3-14B, ground_truth_training=false2026.03 | 76.1 | 76.6 | 67.3 | |
| BaseBackbone=Gemma3-12B-pt, ground_truth_training=false2026.03 | 73.6 | 70.1 | 62.3 | |
| BaseBackbone=Aya-Expanse-8B, ground_truth_training=false2026.03 | 72.2 | 59.8 | 52.9 | |
| BaseBackbone=Qwen2.5-14B, ground_truth_training=false2026.03 | 68.6 | 72.5 | 58.1 | |
| BaseBackbone=Llama3.1-8B, ground_truth_training=false2026.03 | 60.9 | 57.3 | 45.8 | |
| DCOBackbone=Llama3.1-8B, ground_truth_training=true2026.03 | 13.8 | 7.3 | 8.9 | |
| DCOBackbone=Qwen2.5-14B, ground_truth_training=true2026.03 | 13.1 | 7.6 | 13.5 | |
| DPOBackbone=Qwen2.5-14B, ground_truth_training=true2026.03 | 12.3 | 7.8 | 13.9 | |
| DCOBackbone=Qwen2.5-14B, ground_truth_training=false2026.03 | 10.6 | 4 | 9.6 | |
| DCOBackbone=Gemma3-12B-pt, ground_truth_training=true2026.03 | 10.2 | 1.2 | 2.9 | |
| DPOBackbone=Llama3.1-8B, ground_truth_training=true2026.03 | 10.1 | 8 | 8.8 | |
| DCOBackbone=Llama3.1-8B, ground_truth_training=false2026.03 | 9.4 | 7.5 | 7.6 | |
| DPOBackbone=Gemma3-12B-pt, ground_truth_training=true2026.03 | 6.5 | 1.8 | 3.4 | |
| DCOBackbone=Gemma3-12B-pt, ground_truth_training=false2026.03 | 6.5 | 0.9 | 2.5 | |
| DCOBackbone=Aya-Expanse-8B, ground_truth_training=false2026.03 | 5.3 | 0.5 | 0.5 | |
| DCOBackbone=Qwen3-14B, ground_truth_training=true2026.03 | 4.4 | 2.8 | 4.3 | |
| SFTBackbone=Llama3.1-8B, ground_truth_training=true2026.03 | 4.3 | 6.7 | 5.9 | |
| CALMBackbone=Qwen2.5-14B, ground_truth_training=false2026.03 | 4.2 | 0 | 4.1 | |
| SFTBackbone=Aya-Expanse-8B, ground_truth_training=true2026.03 | 3.5 | 0.7 | 0.5 | |
| DCOBackbone=Aya-Expanse-8B, ground_truth_training=true2026.03 | 3.1 | 2.7 | 2.6 | |
| CALMBackbone=Gemma3-12B-pt, ground_truth_training=false2026.03 | 3 | -0.4 | 0 | |
| DPOBackbone=Qwen3-14B, ground_truth_training=true2026.03 | 3 | 2.7 | 4.2 | |
| CALMBackbone=Llama3.1-8B, ground_truth_training=false2026.03 | 3 | -2.2 | -5 | |
| DCOBackbone=Qwen3-14B, ground_truth_training=false2026.03 | 2.7 | 0.4 | 1.3 | |
| CALMBackbone=Aya-Expanse-8B, ground_truth_training=false2026.03 | 1.4 | -2.2 | -2.1 | |
| DPOBackbone=Aya-Expanse-8B, ground_truth_training=true2026.03 | 1.3 | 2.5 | 2.5 | |
| SFTBackbone=Qwen2.5-14B, ground_truth_training=true2026.03 | 0.6 | 1.5 | 6.7 | |
| SFTBackbone=Gemma3-12B-pt, ground_truth_training=true2026.03 | 0.6 | 0.7 | 1.6 | |
| CALMBackbone=Qwen3-14B, ground_truth_training=false2026.03 | 0.3 | -2.1 | -1.1 | |
| SFTBackbone=Qwen3-14B, ground_truth_training=true2026.03 | -0.2 | 0.1 | 0.5 |