Multiple Choice STEM Question Answering on OlmoBaseEval MCSTEM
83.4MCSTEM ScoreSeed 36B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Seed 36BModel Category=Open-weight, Number of Parameters=36B2025.12 | 83.4 | 97.3 | 82.8 | 69.6 | 70.1 | 97.1 | |
| Qwen 2.5 32BModel Category=Open-weight, Number of Parameters=32B2025.12 | 82.2 | 97 | 79.7 | 68.8 | 68.4 | 97.1 | |
| Mistral 3.1 24BModel Category=Open-weight, Number of Parameters=24B2025.12 | 81.5 | 96.2 | 76.1 | 68.8 | 70.4 | 96.3 | |
| Gemma 3 27BModel Category=Open-weight, Number of Parameters=27B2025.12 | 80.2 | 95.8 | 74.9 | 64.7 | 68.7 | 96.8 | |
| Llama 3.1 70BModel Category=Open-weight, Number of Parameters=70B2025.12 | 80.1 | 95.2 | 70 | 67.8 | 72.3 | 95.4 | |
| Qwen 3# Parameters=8B, # Train Tokens=36T, Train Compute (10^23 FLOPs)=17.32026.04 | 78.7 | 95.4 | 76.7 | 63.5 | 62 | 96 | |
| Nemo. 3 Nano# Parameters=30B A3B, # Train Tokens=25T, Train Compute (10^23 FLOPs)=5.42026.04 | 78.6 | 94.8 | 73.8 | 64.2 | 64.9 | 95 | |
| Kimi Linear# Parameters=48B A3B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=1.02026.04 | 77.5 | 94.3 | 67.8 | 64.6 | 66.6 | 94 | |
| Marin 32BModel Category=Fully-open, Number of Parameters=32B2025.12 | 75.9 | 93.4 | 68.4 | 61.8 | 60.8 | 95.1 | |
| Falcon H1# Parameters=7B, # Train Tokens=12T, Train Compute (10^23 FLOPs)=5.52026.04 | 75.7 | 94.2 | 73.8 | 59.9 | 55.9 | 94.7 | |
| LLM 360 K2V2 70BModel Category=Fully-open, Number of Parameters=70B2025.12 | 75.6 | 93 | 64.7 | 63.7 | 61.4 | 95.3 | |
| Gemma 2 27BModel Category=Open-weight, Number of Parameters=27B2025.12 | 75.6 | 94.1 | 65.8 | 61.8 | 61 | 95.1 | |
| OLMo 2 32BModel Category=Fully-open, Number of Parameters=32B2025.12 | 75.3 | 94.4 | 64.7 | 60.2 | 62.2 | 95.1 | |
| Olmo 3 32BModel Category=Fully-open, Number of Parameters=32B2025.12 | 74.5 | 94.7 | 70.8 | 57.6 | 53.8 | 95.5 | |
| Nemotron-H# Parameters=8B, # Train Tokens=15T, Train Compute (10^23 FLOPs)=7.22026.04 | 72.4 | 93.3 | 64.2 | 56.2 | 54.5 | 94.1 | |
| Apertus 70BModel Category=Fully-open, Number of Parameters=70B2025.12 | 70 | 90.7 | 57.8 | 55.9 | 52.4 | 93.3 | |
| Olmo Hybrid# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.6, Decoding Settings=--disable-cascade-attn, --enforce-eager2026.04 | 70 | 90.8 | 64.6 | 52.1 | 48.7 | 93.9 | |
| Olmo 3# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.6, Decoding Settings=--disable-cascade-attn, --enforce-eager2026.04 | 66.2 | 89.2 | 59.7 | 48 | 41.6 | 92.8 | |
| Falcon Mamba# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.52026.04 | 64.2 | 85.7 | 52 | 48.9 | 42.5 | 91.9 | |
| Recurr. Gemma# Parameters=9B, # Train Tokens=2T, Train Compute (10^23 FLOPs)=1.02026.04 | 61.6 | 82.6 | 49.6 | 47.3 | 39.4 | 88.9 | |
| Gaperon 24BModel Category=Fully-open, Number of Parameters=24B2025.12 | 56.2 | 72.7 | 45.3 | 42.6 | 35.4 | 84.9 | |
| xLSTM# Parameters=7B, # Train Tokens=2T, Train Compute (10^23 FLOPs)=0.92026.04 | 36.9 | 46.7 | 34.3 | 31.4 | 23.2 | 48.8 |