Long-context language modeling evaluation on HELMET held-out eval
57.61Accuracy (8K Context)Qwen 2.5 32B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen 2.5 32BScale=32B2025.12 | 57.61 | 56.06 | 54.01 | 41.73 | |
| Olmo 3 32BScale=32B2025.12 | 52.11 | 49.36 | 48.6 | 43.15 | |
| Qwen 3 8BScale=7B2025.12 | 51.62 | 49.9 | 47.71 | — | |
| Xiaomi MiMo 7BScale=7B2025.12 | 50.57 | 49.68 | 46.01 | — | |
| Mistral Small 3.1 24BScale=32B2025.12 | 49.41 | 49.71 | 47.46 | 43.34 | |
| Gemma 3 27BScale=32B2025.12 | 49.37 | 49.92 | 50.31 | 48.6 | |
| Qwen 2.5 7BScale=7B2025.12 | 49.26 | 46.25 | 42.99 | 30.47 | |
| Apertus 8BScale=7B2025.12 | 46.09 | 43.71 | 41.26 | 35.12 | |
| Olmo 3 7BScale=7B2025.12 | 45.66 | 43.62 | 41.15 | 36.8 | |
| Llama 3.1 8BScale=7B2025.12 | 45 | 43.48 | 42.44 | 40.18 | |
| Apertus 70BScale=32B2025.12 | 44.72 | 44.6 | 41.07 | 35.67 | |
| IBM Granite 3.3 8BScale=7B2025.12 | 43.19 | 41.63 | 39.31 | 35.74 | |
| Nemotron Nano 9BScale=7B2025.12 | 41.78 | 42.9 | 41.82 | 41.48 |