Multiple Choice Question Answering on Bolmo Evaluation Suite MC STEM 7B
78.8MC STEM Average AccuracyQwen2 7B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Qwen2 7BOpenness=Open-weight, Parameters=7B2025.12 | 78.8 | 95.4 | 76.7 | 63.5 | 62.1 | 96.1 | |
| Qwen 2.5 7BOpenness=Open-weight, Parameters=7B2025.12 | 74.7 | 93.4 | 67.6 | 60.3 | 56.6 | 95.4 | |
| Nemo-Nano 9BOpenness=Open-weight, Parameters=9B2025.12 | 73.5 | 94.1 | 71.1 | 54.5 | 53.5 | 94.3 | |
| Gemma 2 9BOpenness=Open-weight, Parameters=9B2025.12 | 72.8 | 92.7 | 62.8 | 58.9 | 55.4 | 94.4 | |
| MiMo 7BOpenness=Open-weight, Parameters=7B2025.12 | 71.6 | 91.7 | 63.5 | 56.2 | 53 | 93.5 | |
| Llama 3.1 8BOpenness=Open-weight, Parameters=8B2025.12 | 69 | 86.4 | 55.7 | 56.5 | 53.7 | 92.7 | |
| Marin 8BOpenness=Fully-open, Parameters=8B2025.12 | 68.1 | 89.2 | 58.1 | 52.7 | 47.3 | 93.2 | |
| Olmo 3 7BOpenness=Fully-open, Parameters=7B2025.12 | 66.4 | 89.2 | 59.7 | 48.3 | 41.8 | 92.8 | |
| Olmo 3 7B# Parameters (incl. Embed)=7.30B2025.12 | 66.3 | 89.2 | 59.5 | 48.2 | 42 | 92.8 | |
| Apertus 8BOpenness=Fully-open, Parameters=8B2025.12 | 66.3 | 87.9 | 52.4 | 51.7 | 47.6 | 91.9 | |
| Bolmo 7B# Parameters (incl. Embed)=7.63B2025.12 | 65.5 | 88.5 | 57 | 47.8 | 42.4 | 91.9 | |
| Granite 3.3 8BOpenness=Open-weight, Parameters=8B2025.12 | 65 | 86.2 | 55.6 | 49.6 | 43 | 90.8 | |
| OLMo 2 7BOpenness=Fully-open, Parameters=7B2025.12 | 64.6 | 85.7 | 53.2 | 49.2 | 43.8 | 90.9 | |
| TFree-Hat 7B# Parameters (incl. Embed)=7.19B2025.12 | 62.1 | 90 | 55.2 | 51.5 | 20.5 | 93.3 | |
| Gaperon 8BOpenness=Fully-open, Parameters=8B2025.12 | 58 | 77.2 | 43.1 | 44.5 | 36.8 | 88.4 | |
| EvaByte 6.5B# Parameters (incl. Embed)=6.49B2025.12 | 54 | 74.2 | 44.3 | 37.9 | 27.2 | 86.6 | |
| BLT 7B# Parameters (incl. Embed)=10.55B2025.12 | 49 | 65.5 | 41.6 | 37.6 | 22.5 | 77.8 |