Multilingual Language Understanding on MMMLU (Language Specific Averages)
60.5Accuracy (Korean)Task Arithmetic
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Task ArithmeticBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 60.5 | 22 | 37.2 | 39.9 | — | |
| baseBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 60.3 | 17.4 | 37.4 | 38.4 | — | |
| Ties MergingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 60.3 | 24.2 | 37 | 40.5 | — | |
| DAREBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 60.3 | 22.6 | 37 | 40 | — | |
| Layer SwappingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 60 | 25.3 | 36.6 | 40.6 | — | |
| Module-wise SwappingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 60 | 24.5 | 37.7 | 40.7 | — | |
| Layer-wise SwappingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 59.9 | 24.4 | 37.5 | 40.6 | — | |
| Lang-OnlyBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 59.8 | 19 | 37.3 | 38.7 | — | |
| Safety-OnlyBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 59.7 | 25.1 | 37.2 | 40.7 | — | |
| Mixed SFTBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | 59.6 | 20.3 | 37.3 | 39.1 | — | |
| baseBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 44.5 | 28.9 | 38.6 | 37.3 | — | |
| DAREBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 44.4 | 26.1 | 37.6 | 36 | — | |
| Layer SwappingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 43.6 | 25.3 | 37.3 | 35.4 | — | |
| Module-wise SwappingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 43.6 | 27.5 | 37.4 | 36.2 | — | |
| Task ArithmeticBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 43.5 | 26.1 | 37.5 | 35.7 | — | |
| Mixed SFTBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 43.4 | 24.6 | 37.5 | 35.2 | — | |
| Ties MergingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 43.3 | 24.8 | 37.2 | 35.1 | — | |
| Lang-OnlyBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 43.1 | 24.7 | 38.2 | 35.3 | — | |
| Layer-wise SwappingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 42.9 | 27.6 | 37.5 | 36 | — | |
| Safety-OnlyBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | 42.8 | 24.3 | 36.7 | 34.6 | — | |
| Gemma 3 27BModel size=27B, Thinking mode=false2026.07 | — | — | — | — | 70.7 | |
| Gemma 4 12BModel size=12B, Thinking mode=true2026.07 | — | — | — | — | 83.4 | |
| Gemma 4 26B-A4BModel size=26B-A4B, Thinking mode=true2026.07 | — | — | — | — | 86.3 | |
| Gemma 4 31BModel size=31B, Thinking mode=true2026.07 | — | — | — | — | 88.4 | |
| Gemma 4 E2BModel size=E2B, Thinking mode=true2026.07 | — | — | — | — | 67.4 | |
| Gemma 4 E4BModel size=E4B, Thinking mode=true2026.07 | — | — | — | — | 76.6 |