In-context retrieval on RULER 32K
100S-N ScoreHiLS-Attn-HoPE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| HiLS-Attn-HoPEParameters=345M2026.07 | 100 | 95 | 68 | |
| Naive-BSA-HoPEParameters=345M2026.07 | 98 | 77 | 13 | |
| Full-Attn HoPE (extra MLP)Parameters=345M2026.07 | 83 | 19 | 1 | |
| HSA-UltralongParameters=345M2026.07 | 82 | 74 | 15 | |
| Full-Attn HoPEParameters=345M2026.07 | 72 | 11 | 11 | |
| SWA-RoPEParameters=345M2026.07 | 8 | 2 | 7 | |
| NSA-RoPEParameters=345M2026.07 | 5 | 0 | 0 |