Long-context Language Understanding on RULER 16k context length 1.0
82.6MK1 ScoreSoftmax
Evaluation Results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SoftmaxContext Length=16k2026.05 | 82.6 | 28.8 | 10.4 | 22.6 | 25.8 | 100 | 100 | 98.8 | 15 | 23.9 | 9.7 | 19.4 | 20.8 | 42.9 | |
| EntmaxContext Length=16k2026.05 | 77.8 | 1.2 | 9.4 | 37.8 | 22.4 | 100 | 100 | 71.4 | 23 | 49.3 | 35 | 27.6 | 27.7 | 44.8 | |
| Entmax (Gaus.)Context Length=16k, Selection=Gaussian-aware, Sparse Budget=1K-token2026.05 | 75.4 | 2.6 | 6.2 | 37.9 | 21.7 | 100 | 99.8 | 72.8 | 23.5 | 48 | 34.32 | 26.8 | 27 | 44.3 | |
| Entmax (topk)Context Length=16k, Selection=top-k, Sparse Budget=1K-token2026.05 | 73.4 | 8.4 | 1.2 | 33 | 20.6 | 100 | 98.8 | 71.2 | 4.4 | 42.4 | 33.4 | 23.6 | 25.6 | 41.2 | |
| Softmax (topk)Context Length=16k, Selection=top-k, Sparse Budget=1K-token2026.05 | 54.4 | 4 | 0 | 4.3 | 11.2 | 99 | 88.2 | 48.6 | 2.5 | 9.2 | 2.1 | 19 | 16.2 | 27.6 |