In-context Retrieval on RULER 8K
100S-NFull-Attn RoPE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Full-Attn RoPEParameters=345M, Training context length=8K, Extra parameters=None2026.07 | 100 | 97 | 34 | |
| Full-Attn HoPEParameters=345M, Training context length=8K, Extra parameters=None2026.07 | 100 | 98 | 36 | |
| Full-Attn HoPE (extra MLP)Parameters=345M, Training context length=8K, Extra parameters=0.6%2026.07 | 100 | 99 | 39 | |
| Naive-BSA-HoPEParameters=345M, Training context length=8K, Extra parameters=None2026.07 | 100 | 97 | 23 | |
| HiLS-Attn-HoPEParameters=345M, Training context length=8K, Extra parameters=0.6%2026.07 | 100 | 95 | 72 | |
| Dash-Attention-RoPEParameters=345M, Training context length=8K, Extra parameters=0.006%2026.07 | 92 | 67 | 34 | |
| HSA-UltralongParameters=345M, Training context length=8K, Extra parameters=12.7%2026.07 | 87 | 79 | 23 | |
| InfLLM v2*Parameters=345M, Training context length=8K, Extra parameters=0.006%2026.07 | 77 | 39 | 25 | |
| NSA-RoPEParameters=345M, Training context length=8K, Extra parameters=0.2%2026.07 | 49 | 17 | 16 | |
| SWA-RoPEParameters=345M, Training context length=8K, Extra parameters=None2026.07 | 18 | 8 | 22 |