Long-context language understanding on RULER 16K
89.28CWE ScoreFull Cache
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full CacheBackbone=Llama-3.1-8B-Instruct, Cache Budget=Full2026.05 | 89.28 | 90.33 | 99.6 | 100 | 99 | 98.9 | 98.9 | 100 | 100 | 100 | 80.8 | 57.2 | 99.72 | 93.36 | — | |
| Full CacheBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=Full2026.05 | 82.64 | 88.07 | 97.6 | 95 | 77.6 | 88.5 | 88.65 | 94.4 | 96.4 | 99.6 | 72 | 49.8 | 96.04 | 86.64 | — | |
| CriticalKVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 77.66 | 81.33 | 25.2 | 6.2 | 1.4 | 18.9 | 17.8 | 44 | 35.8 | 2.4 | 35.6 | 37.2 | 41.76 | 32.71 | 0 | |
| CriticalKV w. GRKVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 77.4 | 82.13 | 26.6 | 6 | 1.6 | 19.5 | 18.2 | 45.4 | 38 | 2.4 | 35.6 | 36.8 | 41.88 | 33.19 | 8 | |
| SnapKV w. GRKVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 70.38 | 85 | 16.6 | 5.6 | 1.4 | 11.35 | 9.8 | 45.4 | 14.6 | 2.4 | 28.6 | 33.6 | 16.28 | 26.23 | 9 | |
| SnapKVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 68.32 | 84.67 | 14.4 | 5.6 | 1.2 | 10.75 | 9.35 | 46 | 11 | 2.4 | 28.4 | 34.6 | 15.8 | 25.58 | 0 | |
| CriticalKV w. D2OBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 65.64 | 79.07 | 21 | 3 | 0.8 | 17.05 | 15.65 | 46.4 | 35 | 2.6 | 27.4 | 34.4 | 20.6 | 28.35 | 2 | |
| SnapKV w. D2OBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 59.96 | 83.53 | 12.4 | 3.2 | 0.4 | 10.35 | 9.65 | 45.8 | 10.8 | 2.4 | 26 | 31.6 | 7.64 | 23.36 | 1 | |
| CriticalKV w. CaMBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 43.3 | 88.73 | 23.8 | 4.6 | 1 | 15.75 | 14.6 | 43.6 | 29.4 | 2.4 | 36 | 32.2 | 20.4 | 27.37 | 2 | |
| SnapKV w. CaMBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=20%2026.05 | 42.42 | 91.4 | 14 | 4.4 | 1.4 | 9.6 | 9.05 | 22.6 | 10.8 | 2.4 | 30.4 | 32.6 | 6.68 | 21.37 | 3 | |
| CriticalKV w. GRKVBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 42.22 | 78.8 | 91.2 | 15.2 | 3.6 | 90.15 | 81.25 | 95.2 | 99 | 6.2 | 37.6 | 37.4 | 85.8 | 58.74 | 10 | |
| CriticalKVBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 41.9 | 77.07 | 90.4 | 14.6 | 3.6 | 90.3 | 80.9 | 95 | 99 | 5.8 | 37 | 36.6 | 84.88 | 58.23 | 0 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 22.68 | 72.47 | 59 | 12 | 4.6 | 43.2 | 38.85 | 89.8 | 87.2 | 3.2 | 28.6 | 31.8 | 76.08 | 43.81 | 0 | |
| SnapKV w. GRKVBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 20.3 | 75.07 | 65 | 14.2 | 4.8 | 46.65 | 41.65 | 91.2 | 91 | 3 | 28.6 | 32.2 | 77.48 | 45.47 | 10 | |
| CriticalKV w. D2OBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 12.9 | 70.13 | 86.4 | 5.4 | 1.6 | 86.9 | 75.55 | 94.2 | 96.4 | 3.8 | 23.8 | 31 | 80.04 | 51.39 | 0 | |
| CriticalKV w. CaMBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 4.94 | 84 | 80.6 | 8.4 | 3.2 | 73 | 66.85 | 88.6 | 93.8 | 4.6 | 37.4 | 33.6 | 38.16 | 47.47 | 2 | |
| SnapKV w. D2OBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 3.9 | 68.53 | 50.8 | 4 | 1.6 | 36.35 | 29.3 | 91.4 | 81.2 | 3.2 | 21.8 | 29 | 67.64 | 37.59 | 1 | |
| SnapKV w. CaMBackbone=Llama-3.1-8B-Instruct, Cache Budget=20%2026.05 | 3.58 | 83.8 | 45.2 | 9 | 4.2 | 31.9 | 29.2 | 66.8 | 76.2 | 2.8 | 29 | 32.8 | 27.8 | 34.02 | 3 |