Long-context understanding on RULER (dev)
96.1Accuracy (4K Context)Olmo 3 32B
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Olmo 3 32BScale=32B2025.12 | 96.1 | 94.57 | 90.42 | 86.22 | 79.7 | |
| Mistral Small 3.1 24BScale=32B2025.12 | 96.05 | 95.06 | 93.77 | 92.42 | 88.8 | |
| Qwen 2.5 32BScale=32B2025.12 | 96.03 | 94.52 | 95.07 | 92.67 | 80.73 | |
| Qwen 3 8BScale=7B2025.12 | 95.58 | 94.1 | 93.78 | 90.29 | — | |
| Llama 3.1 8BScale=7B2025.12 | 95.56 | 92.76 | 93.13 | 91.43 | 86.88 | |
| Nemotron Nano 9BScale=7B2025.12 | 95.31 | 93.09 | 91.58 | 89.01 | 85.13 | |
| Olmo 3 7BScale=7B2025.12 | 94.89 | 91.21 | 84.14 | 78.79 | 67.96 | |
| Qwen 2.5 7BScale=7B2025.12 | 94.63 | 90.87 | 88.68 | 87.26 | 67.3 | |
| Xiaomi MiMo 7BScale=7B2025.12 | 94.33 | 93.45 | 92.53 | 89.28 | — | |
| IBM Granite 3.3 8BScale=7B2025.12 | 91.98 | 85.69 | 82.7 | 78.13 | 67.62 | |
| Apertus 70BScale=32B2025.12 | 91.52 | 84.26 | 80.54 | 76.82 | 60.33 | |
| Apertus 8BScale=7B2025.12 | 90.47 | 82.48 | 74.43 | 69.05 | 59.89 | |
| Gemma 3 27BScale=32B2025.12 | 84.48 | 84.2 | 85.36 | 87.06 | 84.59 |