Multilingual Language Understanding on MMLU-ProX
79.5AccuracySuper
Evaluation results
| Method | Links | |
|---|---|---|
| SuperPrecision=BF162026.07 | 79.5 | |
| SuperPrecision=NVFP42026.07 | 79 | |
| Qwen3-8B (Thinking)Base Model=Qwen3-8B, Target-CoT setting=true, Thinking mode=true2025.08 | 78.1 | |
| Puzzle-75B-A9BPrecision=BF162026.07 | 77.5 | |
| Puzzle-75B-A9BPrecision=NVFP42026.07 | 76.5 | |
| Qwen-3-30B-A3BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 72 | |
| OpenThoughts3-20kSFT Dataset=OpenThoughts3-20k, Base Model=Qwen3-8B-Base, Target-CoT setting=true, Thinking mode=false2025.08 | 71.1 | |
| Qwen-3-32BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 70.1 | |
| Translated-s1kSFT Dataset=Translated-s1k, Base Model=Qwen3-8B-Base, Target-CoT setting=true, Thinking mode=false2025.08 | 69 | |
| Distilled-s1kSFT Dataset=Distilled-s1k, Base Model=Qwen3-8B-Base, Target-CoT setting=true, Thinking mode=false2025.08 | 69 | |
| Qwen-3-14BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 66.1 | |
| Llama-3.3-70BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 65.7 | |
| Mistral-3.2-24BOpenness=Open-weights, Region=European, Tuning=Instruction-tuned2026.02 | 64.1 | |
| Gemma-3-27BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 60.2 | |
| OLMo-3.1-32BOpenness=Fully-open, Region=Non-European, Tuning=Instruction-tuned2026.02 | 57 | |
| Gemma-3-12BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 53.3 | |
| EuroLLM-22B (new)Openness=Fully-open, Region=European, Tuning=Instruction-tuned2026.02 | 45.3 | |
| OLMo-3-7BOpenness=Fully-open, Region=Non-European, Tuning=Instruction-tuned2026.02 | 41.8 | |
| LANGbackbone=Qwen2.5-7B-Instruct2026.05 | 41 | |
| LC-GRPObackbone=Qwen2.5-7B-Instruct2026.05 | 39.9 | |
| EuroLLM-22B (old)Openness=Fully-open, Region=European, Tuning=Instruction-tuned2026.02 | 37.9 | |
| EuroLLM-9B (new)Openness=Fully-open, Region=European, Tuning=Instruction-tuned2026.02 | 37.7 | |
| Apertus-70BOpenness=Fully-open, Region=European, Tuning=Instruction-tuned2026.02 | 36.5 | |
| Qwen2.5-7B-Instruct2026.05 | 35.9 | |
| Llama-3.1-8BOpenness=Open-weights, Region=Non-European, Tuning=Instruction-tuned2026.02 | 33.4 | |
| Apertus-8BOpenness=Fully-open, Region=European, Tuning=Instruction-tuned2026.02 | 29.5 | |
| EuroLLM-9B (old)Openness=Fully-open, Region=European, Tuning=Instruction-tuned2026.02 | 29 | |
| Vanilla GRPObackbone=Qwen2.5-7B-Instruct2026.05 | 21 |