In-context retrieval on RULER 16K
100S-N ScoreFull-Attn HoPE (extra MLP)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full-Attn HoPE (extra MLP)Parameters=345M, Training context length=8K2026.07 | 100 | 97 | 13 | |
| HiLS-Attn-HoPEParameters=345M, Training context length=8K2026.07 | 100 | 94 | 65 | |
| Full-Attn HoPEParameters=345M, Training context length=8K2026.07 | 99 | 97 | 21 | |
| Naive-BSA-HoPEParameters=345M, Training context length=8K2026.07 | 99 | 89 | 16 | |
| HSA-UltralongParameters=345M, Training context length=8K2026.07 | 92 | 75 | 17 | |
| InfLLM v2*Parameters=345M, Training context length=8K2026.07 | 21 | 0 | 1 | |
| NSA-RoPEParameters=345M, Training context length=8K2026.07 | 6 | 5 | 4 | |
| SWA-RoPEParameters=345M, Training context length=8K2026.07 | 4 | 1 | 11 | |
| Full-Attn RoPEParameters=345M, Training context length=8K2026.07 | 0 | 0 | 0 | |
| Dash-Attention-RoPEParameters=345M, Training context length=8K2026.07 | 0 | 0 | 0 |