Long-context Language Understanding on RULER 4k context length 1.0
84.4Single-Key (mk1) ScoreSoftmax
Evaluation Results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SoftmaxContext Length=4k2026.05 | 84.4 | 39.8 | 10 | 20.4 | 16.3 | 100 | 100 | 100 | 28 | 30.3 | 14.3 | 22.6 | 37.3 | 46.4 | |
| Entmax (Gaus.)Context Length=4k, Selection=Gaussian-aware, Sparse Budget=1K-token2026.05 | 82.2 | 12.8 | 28.6 | 56.1 | 62.4 | 100 | 99.8 | 75.4 | 40.4 | 56.3 | 37.1 | 29.8 | 36.6 | 55.2 | |
| EntmaxContext Length=4k2026.05 | 81.8 | 13 | 29.6 | 56.3 | 60.6 | 100 | 100 | 76.4 | 48.5 | 58.1 | 37.7 | 29.8 | 37.4 | 56.1 | |
| Entmax (topk)Context Length=4k, Selection=top-k, Sparse Budget=1K-token2026.05 | 81.4 | 21.4 | 22.6 | 60.1 | 63.4 | 100 | 100 | 75.6 | 40.4 | 57.9 | 36.8 | 28.4 | 36.4 | 55.7 | |
| Softmax (topk)Context Length=4k, Selection=top-k, Sparse Budget=1K-token2026.05 | 79.6 | 37.8 | 5.2 | 35.2 | 22 | 100 | 100 | 99.8 | 10.8 | 24.6 | 9.5 | 22.4 | 38.8 | 45.1 |