Long-context retrieval and aggregation on RULER 8k (Average Accuracy)
94.24Average AccuracyNestedKV
Evaluation results
| Method | Links | |
|---|---|---|
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 94.24 | |
| Full KVModel=Qwen3-8B, Eviction ratio (r)=0.02026.05 | 93.71 | |
| Full KVModel=Qwen3-4B, Eviction ratio (r)=0.02026.05 | 91.65 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 91.26 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 91.11 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 89.35 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 85.17 | |
| Full KVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.02026.05 | 83.99 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 83.86 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 82.72 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 81.2 | |
| NestedKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 80.81 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 80.57 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 79.89 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 78.71 | |
| NestedKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 78.52 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 78.28 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 77.65 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 76.49 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 74.75 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 74.36 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 74.11 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 73.85 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 72.77 | |
| NestedKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 70.88 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 70.16 | |
| Full KVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.02026.05 | 70.14 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.252026.05 | 69.62 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 68.71 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.252026.05 | 68.37 | |
| KeyDiffModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 67.74 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 67.71 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 67.55 | |
| KeyDiffModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 67.29 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.252026.05 | 62.74 | |
| KeyDiffModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 62.53 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 61.88 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 61.42 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 59.41 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 58.71 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 58 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 56.03 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 55.99 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 55.75 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 55.35 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 54.16 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 53.77 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 52.12 | |
| NestedKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 51.88 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 50.01 | |
| KeyDiffModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 49.55 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 49.16 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.252026.05 | 47.5 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.502026.05 | 45.47 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.502026.05 | 42.95 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.502026.05 | 41.83 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 40.6 | |
| StreamingLLMModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 38.12 | |
| ExpAttnModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 38 | |
| SnapKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 35.42 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 35.15 | |
| StreamingLLMModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 33.99 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 33.28 | |
| SnapKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 33.27 | |
| SnapKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 30.95 | |
| StreamingLLMModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 28.42 | |
| PyramidKVModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 27.76 | |
| ExpAttnModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 26.34 | |
| PyramidKVModel=Llama-3.2-3B-Inst., Eviction ratio (r)=0.752026.05 | 26.22 | |
| PyramidKVModel=Qwen3-4B, Eviction ratio (r)=0.752026.05 | 23.85 | |
| ExpAttnModel=Qwen3-8B, Eviction ratio (r)=0.752026.05 | 23.14 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.502026.05 | 22.53 | |
| StreamingLLMModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 21.99 | |
| SnapKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 19.79 | |
| ExpAttnModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 18.52 | |
| PyramidKVModel=Llama-3.2-1B-Inst., Eviction ratio (r)=0.752026.05 | 14.9 |