Long-context retrieval and aggregation on RULER 32k (Average Accuracy)
89.56Average AccuracyFull KV
Evaluation results
| Method | Links | |
|---|---|---|
| Full KVModel=Qwen3-8B, Eviction ratio (r)=0.02026.05 | 89.56 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 88.75 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 88.15 | |
| Full KVModel=Qwen3-4B, Eviction ratio (r)=0.02026.05 | 85.78 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 85.41 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 85.25 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 85.01 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 82.61 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 82.58 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 82.52 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 78.11 | |
| Full KVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.02026.05 | 77.26 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 76.17 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 75.54 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 75.4 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 75.32 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 75 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 74.88 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 74.85 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 73.85 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 73.11 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 71.91 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 71.28 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 70.89 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 70.32 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 69.15 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 68.62 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 68.36 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 66.98 | |
| Full KVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.02026.05 | 66.9 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 66.04 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 64.41 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 64.4 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 64.2 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 63.68 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 63.38 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 62.6 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 62.55 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 62.42 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 62.33 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 61.64 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 60.29 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 60.27 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 58.98 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 58.93 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 57.06 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 55.77 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 54.63 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 53.35 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 50.97 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 50.93 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 50.5 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 50.47 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 50.25 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 50.23 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 49.88 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 47.96 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 47.87 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 47.21 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 47.14 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 43.4 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 43.22 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 42.76 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 42.17 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 38.18 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 36.82 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 36.09 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 35.61 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 34.81 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 31.7 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 31.65 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 30.63 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 29.17 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 27.99 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 25.7 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 20.59 |