Long-context Language Understanding on RULER 8k context length 1.0
76MK1 ScoreSoftmax
Evaluation Results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SoftmaxContext Length=8k2026.05 | 76 | 35 | 11.2 | 27 | 26.2 | 100 | 100 | 98.8 | 16.4 | 23.2 | 10.4 | 20.8 | 20.7 | 43.5 | |
| EntmaxContext Length=8k2026.05 | 73.6 | 6.2 | 19.2 | 63.5 | 32.8 | 100 | 100 | 82 | 34.3 | 47.1 | 39.2 | 28.6 | 25.8 | 50.2 | |
| Entmax (Gaus.)Context Length=8k, Selection=Gaussian-aware, Sparse Budget=1K-token2026.05 | 73 | 4 | 15.6 | 60.7 | 32.4 | 100 | 100 | 82 | 32.1 | 47.1 | 27 | 26.7 | 39.5 | 49.3 | |
| Entmax (topk)Context Length=8k, Selection=top-k, Sparse Budget=1K-token2026.05 | 72.2 | 18.2 | 5.8 | 61.5 | 39.7 | 100 | 100 | 83.2 | 15 | 40.5 | 38.5 | 27 | 25 | 48.2 | |
| Softmax (topk)Context Length=8k, Selection=top-k, Sparse Budget=1K-token2026.05 | 70.2 | 33 | 1.4 | 29.3 | 25 | 100 | 93.6 | 86.4 | 5.5 | 23.9 | 7 | 19.6 | 15.3 | 39.3 |