Long-context Language Modeling on LongBench
46.23LongBench Average ScoreAdaKV w/ CriticalKV
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| AdaKV w/ CriticalKVBackbone=Mistral-7B, Cache budget=40%2025.02 | 46.23 | — | — | — | — | — | — | — | |
| HeadKV w/ CriticalKVBackbone=Mistral-7B, Cache budget=40%2025.02 | 46.1 | — | — | — | — | — | — | — | |
| SnapKV w/ CriticalKVBackbone=Mistral-7B, Cache budget=40%2025.02 | 45.35 | — | — | — | — | — | — | — | |
| SnapKV w/ OBCache (Joint)Backbone=Mistral-7B, Cache budget=40%2025.02 | 44.23 | — | — | — | — | — | — | — | |
| SnapKVBackbone=Mistral-7B, Cache budget=40%2025.02 | 44.03 | — | — | — | — | — | — | — | |
| Full KVKV Compression=Disabled2026.05 | 19.3 | — | — | — | — | — | — | — | |
| AMS-TOVAKV Compression=Enabled2026.05 | 19.3 | — | — | — | — | — | — | — | |
| AMS-ExpectedKV Compression=Enabled2026.05 | 19.25 | — | — | — | — | — | — | — | |
| StreamingLLMKV Compression=Enabled2026.05 | 19.19 | — | — | — | — | — | — | — | |
| TOVAKV Compression=Enabled2026.05 | 19.12 | — | — | — | — | — | — | — | |
| PyramidKVKV Compression=Enabled2026.05 | 19.02 | — | — | — | — | — | — | — | |
| AdaKV-ExpE2KV Compression=Enabled2026.05 | 17.58 | — | — | — | — | — | — | — | |
| Qwen3-1.7BAttention Mechanism=Global-Attn2026.01 | — | 42.04 | 86.2 | 23.23 | 72.67 | 31.11 | 100 | 59.21 | |
| Qwen3-1.7BAttention Mechanism=Window-Attn, Window Size (W)=20482026.01 | — | 32.51 | 61.56 | 10.43 | 52.67 | 13.08 | 6.2 | 29.41 | |
| Qwen3-1.7BAttention Mechanism=Window-Attn + ROSA2026.01 | — | 40.53 | 84.34 | 23.76 | 68 | 26.19 | 100 | 57.14 |