Long-context Language Modeling on RULER (4K Context)
95.3AccuracyFull KV
Evaluation Results
| Method | Links | |
|---|---|---|
| Full KVBackbone=LLaMA-3.1-8B, KV Budget (%)=1002025.10 | 95.3 | |
| H2O + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=402025.10 | 82 | |
| H2O + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=402025.10 | 81.3 | |
| TOVA + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 76.7 | |
| H2O + OBCACHE-VBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=402025.10 | 76 | |
| H2O + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=302025.10 | 75.6 | |
| H2O + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=302025.10 | 75.4 | |
| H2OBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=402025.10 | 75.3 | |
| TOVABackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 74.5 | |
| H2O + OBCACHE-VBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=302025.10 | 68.6 | |
| H2O + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 67.8 | |
| H2O + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 67.6 | |
| H2O + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=202025.10 | 67.5 | |
| H2OBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=302025.10 | 67.1 | |
| H2O + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=202025.10 | 65.9 | |
| TOVA + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=102025.10 | 65.5 | |
| TOVABackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=102025.10 | 61.8 | |
| H2O + OBCACHE-VBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 59.9 | |
| H2OBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=Avg.2025.10 | 57.5 | |
| H2O + OBCACHE-VBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=202025.10 | 57.1 | |
| AdaKV + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Agnostic, KV Budget (%)=Avg.2025.10 | 55.2 | |
| H2OBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=202025.10 | 54.2 | |
| H2O + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=102025.10 | 47.7 | |
| H2O + OBCACHE-V&KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=102025.10 | 46.3 | |
| H2O + OBCACHE-VBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=102025.10 | 37.7 | |
| H2OBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Aware, KV Budget (%)=102025.10 | 33.5 | |
| SnapKV + OBCACHE-KBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Agnostic, KV Budget (%)=102025.10 | 24.1 | |
| AdaKVBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Agnostic, KV Budget (%)=102025.10 | 21.5 | |
| SnapKVBackbone=LLaMA-3.1-8B, Evaluation Protocol=Q-Agnostic, KV Budget (%)=102025.10 | 19.4 |