Reasoning and Knowledge Suite (MMLU, ARC, BoolQ, CSQA, HSwag, PIQA, SocIQ, Wino)
75.78MMLUQwen3-4B
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-4BParameters=4B, Shot setting=5 shot2025.12 | 75.78 | 89.83 | 97.53 | 86.09 | 81.9 | 79.46 | 84.98 | 75.59 | 65.43 | 81.84 | |
| Qwen2.5-3BParameters=3B, Shot setting=5 shot2025.12 | 66.86 | 86.44 | 92.59 | 83.88 | 76.09 | 73.85 | 81.45 | 69.4 | 63.69 | 77.14 | |
| Qwen3-1.7BParameters=1.7B, Shot setting=5 shot2025.12 | 65.35 | 80.34 | 91.89 | 79.82 | 74.61 | 60.76 | 77.2 | 68.58 | 59.27 | 73.09 | |
| SmolLM3-3BParameters=3B, Shot setting=5 shot2025.12 | 63.04 | 77.29 | 88.54 | 76.12 | 70.52 | 69.2 | 79.05 | 65.25 | 64.4 | 72.6 | |
| Qwen2.5-1.5BParameters=1.5B, Shot setting=5 shot2025.12 | 61.56 | 79.32 | 90.48 | 76.39 | 75.1 | 64.18 | 76.17 | 64.94 | 59.67 | 71.98 | |
| Llama-3.2-3BParameters=3B, Shot setting=5 shot2025.12 | 57.87 | 72.2 | 83.95 | 76.73 | 70.35 | 71.06 | 79.05 | 64.33 | 64.09 | 71.07 | |
| Qwen2-1.5BParameters=1.5B, Shot setting=5 shot2025.12 | 56.36 | 70.17 | 83.6 | 71.9 | 70.52 | 60.77 | 75.73 | 63.46 | 59.83 | 68.04 | |
| Gemma2-2BParameters=2B, Shot setting=5 shot2025.12 | 55.2 | 66.44 | 82.54 | 72.42 | 69.45 | 66.2 | 78.89 | 65.92 | 65.35 | 69.16 | |
| Qwen3-0.6BParameters=0.6B, Shot setting=5 shot2025.12 | 55.09 | 68.14 | 84.48 | 69.05 | 61.18 | 48.51 | 69.97 | 61.51 | 55.64 | 63.73 | |
| PCMind-2.1-Kaiyuan-2BParameters=2B, Shot setting=5 shot2025.12 | 53.9 | 66.1 | 82.89 | 78.53 | 67.4 | 58.13 | 74.37 | 62.59 | 65.75 | 67.74 | |
| SmolLM2-1.7BParameters=1.7B, Shot setting=5 shot2025.12 | 51.99 | 59.66 | 82.72 | 69.85 | 67.16 | 65.3 | 78.51 | 60.18 | 59.12 | 66.05 | |
| YuLan-Mini-2.4BParameters=2.4B, Shot setting=5 shot2025.12 | 51.76 | 64.75 | 82.54 | 78.59 | 66.18 | 61.2 | 77.31 | 63.25 | 61.88 | 67.5 | |
| OLMo-2-0425-1BParameters=1B, Shot setting=5 shot2025.12 | 44.25 | 47.46 | 76.72 | 70.55 | 65.6 | 61.61 | 76.44 | 55.53 | 60.38 | 62.06 | |
| Llama-3.2-1BParameters=1B, Shot setting=5 shot2025.12 | 37.74 | 36.95 | 70.55 | 67.43 | 62.82 | 60.2 | 74.92 | 50.61 | 58.17 | 57.71 |