Long-context language modeling evaluation on RULER 32K
88.6Average Score (RULER 32K)Full KV
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Full KVBackbone=LLaMA-3.1-8B, KV Budget (%)=1002025.10 | 88.6 | — | — | — | — | — | — | — | — | — | |
| AdaKV + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=402025.10 | 88.5 | — | — | — | — | — | — | — | — | — | |
| AdaKV + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 85.2 | — | — | — | — | — | — | — | — | — | |
| FullContext Length=32K2026.03 | 75.4 | 99.6 | 98 | 98 | 97 | 46.6 | 94.15 | 22.28 | 69.78 | 53.2 | |
| AdaKV + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Agnostic, KV Budget (%)=402025.10 | 75.4 | — | — | — | — | — | — | — | — | — | |
| MixedDimKV-HContext Length=32K2026.03 | 62.64 | 100 | 95.2 | 96 | 38.4 | 31.05 | 64.35 | 21.6 | 65.8 | 51.4 | |
| HeadKVContext Length=32K2026.03 | 60.43 | 100 | 90.6 | 92 | 44 | 24.8 | 56.5 | 20.92 | 64.27 | 50.8 | |
| MixedDimKVContext Length=32K2026.03 | 49.84 | 98.4 | 85.6 | 92.4 | 8.4 | 7.75 | 17.1 | 21.24 | 66.47 | 51.2 | |
| SnapKVContext Length=32K2026.03 | 49.16 | 100 | 85.8 | 89.8 | 13.6 | 7.25 | 16.5 | 20.88 | 62.77 | 45.8 | |
| PyramidKVContext Length=32K2026.03 | 45.31 | 100 | 80.6 | 78.8 | 14.4 | 3.69 | 5.05 | 20.48 | 60.27 | 44.6 | |
| H2OBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Agnostic, KV Budget (%)=102025.10 | 21.1 | — | — | — | — | — | — | — | — | — | |
| H2OContext Length=32K2026.03 | 5.85 | 0.8 | 0 | 0 | 0 | 0 | 0 | 4.04 | 24.2 | 23.6 |