Long-context language understanding on RULER 16K 1.0 (test)
89.28CWE ScoreFull Cache
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full CacheBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=100%2026.05 | 89.28 | 90.33 | 99.6 | 100 | 99 | 98.9 | 98.9 | 100 | 100 | 100 | 80.8 | 57.2 | 99.72 | 93.36 | — | |
| Full CacheBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=100%2026.05 | 82.64 | 88.07 | 97.6 | 95 | 77.6 | 88.5 | 88.65 | 94.4 | 96.4 | 99.6 | 72 | 49.8 | 96.04 | 86.64 | — | |
| CriticalKV w. GRKVBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 59.84 | 79.33 | 13.6 | 2.8 | 0 | 10.15 | 9.75 | 39.2 | 9.6 | 2.4 | 28 | 31.6 | 14 | 23.1 | 10 | |
| CriticalKVBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 59.1 | 78.2 | 13 | 2.4 | 0 | 10.05 | 9.45 | 36.8 | 9.2 | 2.4 | 28.2 | 31.4 | 13.24 | 22.57 | 0 | |
| SnapKV w. GRKVBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 47.32 | 83.33 | 10.8 | 2 | 0.4 | 9.8 | 9.7 | 42.2 | 4.4 | 2.4 | 24.6 | 30.4 | 14.32 | 21.67 | 10 | |
| SnapKVBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 45.98 | 82.8 | 10.4 | 1.8 | 0.4 | 9.6 | 9.5 | 40.8 | 3.4 | 2.4 | 24.2 | 31.8 | 12.32 | 21.18 | 0 | |
| CriticalKV w. D2OBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 30.74 | 71.33 | 9.8 | 0.2 | 0 | 8.7 | 9.5 | 40.4 | 5.8 | 2.4 | 22 | 29 | 4.56 | 18.03 | 2 | |
| SnapKV w. D2OBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 22.78 | 73.13 | 8.2 | 0 | 0 | 8.7 | 9.2 | 43.2 | 4 | 2.4 | 21.6 | 28.6 | 2.68 | 17.27 | 2 | |
| SnapKV w. CaMBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 10.32 | 64.73 | 10.2 | 1.4 | 0 | 9.05 | 9.2 | 5.4 | 4 | 2.4 | 25.4 | 27.4 | 1.44 | 13.15 | 2 | |
| CriticalKVBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 9.72 | 58.73 | 50.2 | 5.6 | 1.6 | 45.1 | 46.3 | 93.6 | 94.4 | 2.6 | 23.8 | 30.4 | 68.72 | 40.83 | 0 | |
| CriticalKV w. GRKVBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 8.8 | 64.93 | 50 | 5.8 | 1.6 | 45.6 | 46.65 | 93.6 | 95.4 | 2.6 | 24 | 31 | 69.6 | 41.51 | 8 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 3.24 | 59 | 23.2 | 4 | 1.4 | 17 | 15.1 | 78.4 | 54.2 | 2.4 | 21 | 28 | 49.72 | 27.44 | 0 | |
| CriticalKV w. CaMBackbone=Mistral-7B-Instruct-v0.3, Context Length=16K, Cache Budget=10%2026.05 | 2.06 | 19.8 | 11.8 | 0.2 | 0 | 9.15 | 9.2 | 0 | 5.6 | 2.4 | 26 | 26.8 | 0 | 8.69 | 0 | |
| SnapKV w. GRKVBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 1.78 | 66.4 | 25.6 | 4.4 | 1.4 | 17.6 | 15.8 | 82.4 | 58.2 | 2.4 | 20.2 | 27.8 | 54.24 | 29.09 | 8 | |
| CriticalKV w. D2OBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 0.28 | 53.8 | 36.4 | 1 | 0 | 36.95 | 32.45 | 93.8 | 80.2 | 2.4 | 15 | 26.2 | 52.28 | 33.14 | 1 | |
| SnapKV w. CaMBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 0.12 | 56.2 | 17.6 | 2.2 | 1.6 | 13.95 | 13.2 | 41.4 | 31.2 | 2.4 | 23 | 28.2 | 5.44 | 18.19 | 3 | |
| CriticalKV w. CaMBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 0.1 | 57.87 | 34.2 | 2.2 | 1.2 | 28.65 | 28.05 | 50.6 | 71 | 2.6 | 25.6 | 25.8 | 6.48 | 25.72 | 1 | |
| SnapKV w. D2OBackbone=Llama-3.1-8B-Instruct, Context Length=16K, Cache Budget=10%2026.05 | 0.02 | 57.8 | 16.4 | 0.4 | 0.2 | 12.7 | 11.25 | 79.8 | 47.6 | 2.4 | 14.6 | 23.8 | 36.6 | 23.35 | 1 |