Long-context language understanding on LongBench <8K context v1
37.5SDoc AccuracyHiLS-Attn
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| HiLS-AttnPositional Embedding=NoPE2026.07 | 37.5 | 34.3 | 23.6 | 64.4 | 5.4 | 62.3 | |
| Olmo3-512swa-CPT2026.07 | 37.4 | 27.8 | 22.1 | 64.1 | 4.1 | 59.4 | |
| HiLS-AttnPositional Embedding=RoPE2026.07 | 37.2 | 33.1 | 22.2 | 61 | 5.4 | 60.6 | |
| Olmo3-Base2026.07 | 36.9 | 33.9 | 23.4 | 63.8 | 5.8 | 61.8 | |
| HiLS-AttnPositional Embedding=HoPE2026.07 | 36.4 | 37.8 | 24.6 | 64.6 | 4.7 | 59.3 | |
| Olmo3-512swa-CPTExtension=YaRN 32K2026.07 | 34.1 | 30.6 | 21.5 | 64 | 3.5 | 60 | |
| LDM token tuning2026.07 | 32.1 | 30.4 | 20.6 | 62.2 | 4.1 | 60.9 |