Long-context evaluation on LongBench (test)
32.85NarQA ScorexKV
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| xKVBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.032025.03 | 32.85 | 45.62 | 54.69 | 51.74 | 36.38 | 28.33 | 31.32 | 24.49 | 22.61 | 26.26 | 100 | 41.3 | |
| PyramidKVBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.002025.03 | 31.83 | 44.36 | 52.56 | 56.38 | 48.35 | 27.23 | 28.63 | 25.06 | 23.49 | 24.8 | 99.5 | 42.02 | |
| TokenButlerBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.002025.03 | 30.73 | 48.86 | 56.45 | 57.77 | 50.82 | 28.34 | 34.34 | 25.3 | 26.91 | 29.22 | 100 | 44.43 | |
| DenseBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=1.002025.03 | 30.64 | 46.64 | 55.42 | 58.17 | 47.76 | 28.23 | 34.51 | 25.61 | 25.43 | 23.51 | 100 | 43.27 | |
| SnapKVBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.002025.03 | 30.34 | 45.56 | 53.62 | 57.92 | 47.5 | 26.96 | 28.99 | 25.33 | 23.95 | 24.91 | 99.5 | 42.23 | |
| PyramidKVBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=8.002025.03 | 29.93 | 41.29 | 47.21 | 60.16 | 51.98 | 32.79 | 27.21 | 22.88 | 19.86 | 45 | 100 | 43.48 | |
| KIVIBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=7.102025.03 | 29.84 | 41.49 | 51.28 | 57.27 | 42.54 | 27.54 | 33.28 | 25.01 | 23.9 | 25.88 | 98.17 | 41.47 | |
| TokenButlerBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=8.002025.03 | 29.39 | 47.27 | 52.5 | 59.67 | 56.01 | 34.58 | 33.32 | 23.14 | 24.04 | 35.09 | 100 | 45 | |
| DenseBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=1.002025.03 | 29.21 | 43.78 | 48.58 | 60.92 | 53.29 | 33.68 | 33.23 | 23.24 | 23.53 | 43.21 | 100 | 44.79 | |
| xKVBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=8.032025.03 | 28.78 | 42.98 | 47.43 | 58.79 | 52.42 | 32.69 | 33.02 | 23.67 | 23.18 | 38.62 | 98 | 43.6 | |
| SnapKVBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=8.002025.03 | 28.68 | 44.51 | 47.43 | 60.88 | 51.91 | 32.17 | 29.95 | 23.05 | 21.79 | 45.67 | 100 | 44.19 | |
| Single SVDBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=8.402025.03 | 27.88 | 41.84 | 47.66 | 52.73 | 48.76 | 28.46 | 28.71 | 22.24 | 21.18 | 33.18 | 65 | 37.97 | |
| KIVIBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=7.102025.03 | 26.43 | 35.78 | 38.4 | 46.17 | 45.08 | 23.79 | 24.62 | 22.41 | 16.98 | 39.52 | 63.5 | 34.79 | |
| StreamingLLMBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.002025.03 | 26.25 | 29.32 | 34.56 | 49.99 | 42.47 | 20.92 | 26.29 | 21.9 | 22.63 | 23.87 | 93.5 | 35.61 | |
| StreamingLLMBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=8.002025.03 | 24.13 | 34.99 | 27.47 | 47.55 | 45.13 | 21.44 | 26.76 | 19.34 | 21.15 | 44.11 | 29 | 31.01 | |
| MiniCacheBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=1.302025.03 | 18.11 | 20.41 | 27.51 | 38.8 | 21.37 | 20.19 | 21.09 | 20.85 | 19.6 | 22.75 | 98 | 29.88 | |
| Single SVDBase Model=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.402025.03 | 15.52 | 38.36 | 37.14 | 30.04 | 19.7 | 22.06 | 19.52 | 22.35 | 18.91 | 22.82 | 85 | 30.13 | |
| MiniCacheBase Model=Qwen2.5-7B-Instruct-1M, K.V. BW (x ↓)=1.302025.03 | 7.98 | 16.47 | 9.42 | 21.78 | 15.46 | 10.55 | 8.83 | 7.94 | 3.84 | 11.32 | 5 | 10.78 |