Long-context modeling on RULER
90.2NIAH Score (MK, 1K)Transformer
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TransformerTraining context length=2K2025.10 | 90.2 | 63.2 | 0.4 | 81.8 | 64.3 | 13.1 | 42.9 | 20.9 | 0 | 21.3 | 2.8 | 0.6 | — | — | |
| NHATraining context length=2K2025.10 | 81.4 | 51.8 | 21.6 | 62.7 | 50.6 | 28.4 | 63.8 | 33.2 | 9.3 | 27.4 | 26 | 24.8 | — | — | |
| GDN-HTraining context length=2K2025.10 | 59.6 | 29.4 | 15 | 56.1 | 41.4 | 27.1 | 64.7 | 31 | 4.1 | 26.5 | 21.6 | 16.8 | — | — | |
| GLA-HTraining context length=2K2025.10 | 46 | 23.6 | 12.4 | 37.9 | 30.1 | 25 | 50.4 | 12.7 | 10.4 | 26.2 | 20.6 | 19.2 | — | — | |
| GSA-HTraining context length=2K2025.10 | 20 | 11.2 | 6.4 | 77.7 | 68.5 | 34.4 | 42.1 | 24.4 | 15.4 | 24.8 | 20.6 | 18 | — | — | |
| Mamba2-HTraining context length=2K2025.10 | 13.6 | 3.8 | 4.8 | 52.5 | 55.8 | 35 | 14.5 | 9 | 8.5 | 4.4 | 19.4 | 16 | — | — | |
| S2: S1 → SFT(2) 60B Targeted (all-attention)Campaign=S2: S1 → SFT(2) 60B Targeted 8 placements, Config=all-attention, @32k speedup=1.0×2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | 79.4 | 82.6 | |
| S2: S1 → SFT(2) 60B Targeted (Reg|Lklhd–26)Campaign=S2: S1 → SFT(2) 60B Targeted 8 placements, Config=Reg|Lklhd–26, @32k speedup=2.9×2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | 74.4 | 82 |