Long-context retrieval and aggregation on RULER 4k
94.73Average AccuracyNestedKV
Evaluation results
| Method | Links | |
|---|---|---|
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 94.73 | |
| Full KVModel=Qwen3-8B, Eviction ratio (r)=0.02026.05 | 94.52 | |
| Full KVModel=Qwen3-4B, Eviction ratio (r)=0.02026.05 | 93.83 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 93.6 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 93.24 | |
| Full KVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.02026.05 | 91.8 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 90.4 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 90.16 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 89.12 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 88.89 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 88.69 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 85.48 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 85.02 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 84 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 82.32 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 80.66 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 80.61 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 80.59 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 79.48 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 79.32 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 79.11 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 77.78 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 77.72 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 77.21 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 76.9 | |
| Full KVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.02026.05 | 75.5 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 75.37 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 75.37 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 75.06 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 73.84 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 73.16 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 72.39 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 72.28 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 70.09 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 68.74 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 66.26 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 65.53 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 65.5 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 63.03 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 61.48 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 61.34 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 60.97 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 60.53 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 60.35 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 60.22 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 59.53 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 58.23 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 56.7 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 56.55 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 56.33 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 52.64 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 50.26 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 49.73 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 43.13 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 42.9 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 42.8 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 42.24 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 40.48 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 40.18 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 39.95 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 37.53 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 34.45 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 33.09 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 32.09 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 29.49 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 28.42 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 28.37 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 28.19 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 25.72 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 24.76 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 22.64 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 22.18 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 21.48 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 20.78 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 20.67 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 19.15 |