Mathematical Reasoning on OlmoBaseEval Math (GSM8k, GSM Symbolic, MATH)
68.5Math Aggregate ScoreKimi Linear
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Kimi Linear# Parameters=48B A3B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=1.02026.04 | 68.5 | 84.9 | 66.7 | 54 | |
| Qwen2 7BOpenness=Open-weight, Parameters=7B2025.12 | 67.2 | 84.5 | 65.4 | 51.6 | |
| Qwen 3# Parameters=8B, # Train Tokens=36T, Train Compute (10^23 FLOPs)=17.32026.04 | 67.2 | 84.2 | 65.4 | 52 | |
| Falcon H1# Parameters=7B, # Train Tokens=12T, Train Compute (10^23 FLOPs)=5.52026.04 | 65.7 | 80.9 | 64.8 | 51.4 | |
| Qwen 2.5 32BModel Category=Open-weight, Number of Parameters=32B2025.12 | 64.7 | 81.1 | 56.2 | 56.7 | |
| Gemma 3 27BModel Category=Open-weight, Number of Parameters=27B2025.12 | 63.2 | 81.3 | 61.2 | 47 | |
| Llama 3.1 70BModel Category=Open-weight, Number of Parameters=70B2025.12 | 62 | 81.2 | 64.6 | 40.2 | |
| Olmo 3 32BModel Category=Fully-open, Number of Parameters=32B2025.12 | 61.9 | 80.6 | 61.2 | 43.8 | |
| Qwen 2.5 7BOpenness=Open-weight, Parameters=7B2025.12 | 60.7 | 79.9 | 56.2 | 45.9 | |
| Mistral 3.1 24BModel Category=Open-weight, Number of Parameters=24B2025.12 | 59.5 | 79.3 | 59.1 | 40.1 | |
| Gemma 2 27BModel Category=Open-weight, Number of Parameters=27B2025.12 | 57.5 | 76.3 | 57.3 | 38.8 | |
| Olmo Hybrid# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.6, Decoding Settings=--disable-cascade-attn, --enforce-eager2026.04 | 55.1 | 74.3 | 49.2 | 41.8 | |
| Olmo 3 7BOpenness=Fully-open, Parameters=7B2025.12 | 54.7 | 75.5 | 48.6 | 40 | |
| Nemotron-H# Parameters=8B, # Train Tokens=15T, Train Compute (10^23 FLOPs)=7.22026.04 | 54.6 | 76.8 | 55.4 | 31.7 | |
| Olmo 3# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.6, Decoding Settings=--disable-cascade-attn, --enforce-eager2026.04 | 54.6 | 75.2 | 48.4 | 40.1 | |
| MiMo 7BOpenness=Open-weight, Parameters=7B2025.12 | 54.3 | 74.3 | 53.3 | 35.2 | |
| OLMo 2 32BModel Category=Fully-open, Number of Parameters=32B2025.12 | 53.9 | 77.6 | 53.1 | 31 | |
| Nemo. 3 Nano# Parameters=30B A3B, # Train Tokens=25T, Train Compute (10^23 FLOPs)=5.42026.04 | 53.2 | 86.3 | 68.6 | 4.6 | |
| Nemo-Nano 9BOpenness=Open-weight, Parameters=9B2025.12 | 49.8 | 82.3 | 62.7 | 4.5 | |
| Marin 32BModel Category=Fully-open, Number of Parameters=32B2025.12 | 49.3 | 69.1 | 42 | 36.8 | |
| Gemma 2 9BOpenness=Open-weight, Parameters=9B2025.12 | 48.8 | 68.5 | 45.1 | 32.9 | |
| LLM 360 K2V2 70BModel Category=Fully-open, Number of Parameters=70B2025.12 | 46.2 | 66.7 | 44.4 | 27.4 | |
| OLMo 2 7BOpenness=Fully-open, Parameters=7B2025.12 | 41.7 | 67.1 | 38.8 | 19.1 | |
| Granite 3.3 8BOpenness=Open-weight, Parameters=8B2025.12 | 41.5 | 61 | 35.5 | 27.9 | |
| Apertus 70BModel Category=Fully-open, Number of Parameters=70B2025.12 | 39.7 | 63 | 38.6 | 17.4 | |
| Marin 8BOpenness=Fully-open, Parameters=8B2025.12 | 39.6 | 60.9 | 33.6 | 24.3 | |
| Llama 3.1 8BOpenness=Open-weight, Parameters=8B2025.12 | 36.9 | 56.4 | 35.1 | 19.2 | |
| Falcon Mamba# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.52026.04 | 33.7 | 57.1 | 25.8 | 18.2 | |
| Recurr. Gemma# Parameters=9B, # Train Tokens=2T, Train Compute (10^23 FLOPs)=1.02026.04 | 32.1 | 49.6 | 25.3 | 21.3 | |
| Apertus 8BOpenness=Fully-open, Parameters=8B2025.12 | 29.2 | 48.2 | 26.3 | 13.1 | |
| Gaperon 24BModel Category=Fully-open, Number of Parameters=24B2025.12 | 20.7 | 33.3 | 14.5 | 14.2 | |
| xLSTM# Parameters=7B, # Train Tokens=2T, Train Compute (10^23 FLOPs)=0.92026.04 | 18.3 | 32.8 | 10.8 | 11.3 | |
| Gaperon 8BOpenness=Fully-open, Parameters=8B2025.12 | 16.9 | 30 | 12.5 | 8.2 | |
| Seed 36BModel Category=Open-weight, Number of Parameters=36B2025.12 | 15.3 | 26.9 | 10.3 | 8.7 |