Long-context retrieval and reasoning on RULER 8k context
100Score 1 (S1)Transformer
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Transformer2026.03 | 100 | 100 | 95 | 87 | 37 | 41 | 44 | 0.6 | 37 | 28 | 27 | — | |
| GDN2026.03 | 100 | 66 | 24 | 31 | 0.4 | 24 | 22 | 6.7 | 11 | 20 | 12 | — | |
| GDN-GSAKV cache ratio=50%2026.03 | 100 | 100 | 86 | 86 | 12 | 62 | 57 | 0.2 | 12 | 31 | 23 | — | |
| HAM Fixed τKV cache ratio=50%, Threshold type=Fixed2026.03 | 100 | 94 | 12 | 28 | 0.4 | 22 | 14 | 37 | 28 | 22 | 16 | — | |
| HAM Learned τKV cache ratio=50%, Threshold type=Learned2026.03 | 100 | 90 | 4.6 | 38 | 0.2 | 22 | 26 | 0.5 | 32 | 25 | 16 | — | |
| HAM Learned routerKV cache ratio=50%, Routing mechanism=Learned router2026.03 | 100 | 100 | 60 | 90 | 45 | 64 | 74 | 26 | 1.1 | 26 | 28 | — | |
| HAM Learned router + EDAKV cache ratio=50%, Routing mechanism=Learned router, Exponential decay averaging (EDA)=true2026.03 | 100 | 100 | 37 | 79 | 18 | 33 | 29 | 0.9 | 2.6 | 22 | 26 | — | |
| Full AttentionSetting=CPT setting2026.04 | — | — | — | — | — | — | — | — | — | — | — | 91.38 | |
| Hybrid-KSASetting=CPT setting2026.04 | — | — | — | — | — | — | — | — | — | — | — | 90.53 | |
| Hybrid-LinearSetting=CPT setting2026.04 | — | — | — | — | — | — | — | — | — | — | — | 83.86 | |
| Hybrid-SCASetting=CPT setting2026.04 | — | — | — | — | — | — | — | — | — | — | — | 84.28 | |
| Hybrid-SWASetting=CPT setting2026.04 | — | — | — | — | — | — | — | — | — | — | — | 88.03 | |
| KSASetting=CPT setting2026.04 | — | — | — | — | — | — | — | — | — | — | — | 86.78 |