Long-context retrieval and aggregation on RULER 16k
93.07Average AccuracyFull KV
Evaluation results
| Method | Links | |
|---|---|---|
| Full KVModel=Qwen3-8B, Eviction ratio (r)=0.02026.05 | 93.07 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 91.46 | |
| Full KVModel=Qwen3-4B, Eviction ratio (r)=0.02026.05 | 91.13 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 90.44 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 88.36 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 88.1 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 84.65 | |
| Full KVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.02026.05 | 84.04 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 82.67 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 82.19 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 81.68 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 81.05 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 79.89 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 79.3 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 77.8 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 77.05 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 76.84 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 76.83 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 76.6 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 76.54 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 74.73 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 73.57 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 73.47 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 73 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 72.51 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 70 | |
| Full KVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.02026.05 | 69.19 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 69.13 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 68.84 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 66.99 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 66.63 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 66.49 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 66.46 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 65.26 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 65.22 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 63.34 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 62.48 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 62.2 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 61.73 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 60.79 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 60 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 55.04 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 54.55 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 54.2 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 53.87 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 52.38 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 52.15 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 50.56 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 50.2 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 49.75 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 49.74 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 49.47 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 46.9 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 46.82 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 45.91 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 45.28 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 44.92 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 39.64 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 37.95 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 37.47 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 37.41 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 35.85 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 35.81 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 34.98 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 34.2 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 31.25 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 30.81 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 30.44 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 29.76 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 29.51 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 27.23 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 24.23 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 22.72 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 22.48 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 16.8 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 14.5 |