Long-context Evaluation on Ruler (test)
100S-NIAH-1FullKV
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FullKVLLM=Qwen3-8B, KV Cache Size=Full2026.03 | 100 | 100 | 100 | 99.6 | 99.6 | 99.6 | 99.9 | 99.75 | 83.98 | 90.67 | 100 | 97.55 | |
| H2OLLM=Qwen3-8B, KV Cache Size=40962026.03 | 100 | 99.8 | 22 | 99.8 | 84.8 | 32.4 | 99.5 | 89.9 | 46.74 | 93 | 99.92 | 78.9 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=40962026.03 | 100 | 100 | 24.8 | 99.6 | 91 | 51.2 | 99.9 | 99.55 | 57.4 | 92.87 | 100 | 83.3 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=40962026.03 | 100 | 100 | 75.2 | 99.8 | 96 | 58 | 99.9 | 99.75 | 72.7 | 93.27 | 100 | 90.42 | |
| DapQLLM=Qwen3-8B, KV Cache Size=40962026.03 | 100 | 100 | 96.4 | 99.8 | 93 | 58.4 | 99.9 | 99.85 | 71.82 | 93.6 | 100 | 92.07 | |
| H2OLLM=Qwen3-8B, KV Cache Size=20482026.03 | 100 | 99.4 | 7.8 | 97 | 64.8 | 10.6 | 94.7 | 48.3 | 28.48 | 94.53 | 99.68 | 67.75 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=20482026.03 | 100 | 100 | 1.6 | 100 | 79.4 | 19 | 99.9 | 93.6 | 35.46 | 95.6 | 100 | 74.96 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=20482026.03 | 100 | 100 | 20 | 100 | 91.6 | 30.4 | 99.85 | 97.75 | 49.59 | 96 | 100 | 80.47 | |
| DapQLLM=Qwen3-8B, KV Cache Size=20482026.03 | 100 | 100 | 55 | 100 | 88.05 | 31 | 99.95 | 95.15 | 45.76 | 96.07 | 100 | 82.82 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=10242026.03 | 100 | 99.8 | 0 | 98.6 | 61.8 | 2.4 | 97.75 | 70.2 | 17.84 | 92.47 | 99.12 | 67.27 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=10242026.03 | 100 | 99.6 | 1 | 99.6 | 79.2 | 13 | 99.25 | 83.05 | 26.52 | 96.27 | 98.84 | 72.39 | |
| DapQLLM=Qwen3-8B, KV Cache Size=10242026.03 | 100 | 99.8 | 14.2 | 99.6 | 75.4 | 14.8 | 99.75 | 78.25 | 23.78 | 97.73 | 99.12 | 72.95 | |
| DapQLLM=Qwen3-8B, KV Cache Size=5122026.03 | 100 | 99.6 | 5.8 | 95.4 | 68.6 | 5.2 | 94.85 | 42.45 | 12.38 | 96.27 | 96.08 | 65.15 | |
| DapQLLM=Qwen3-8B, KV Cache Size=2562026.03 | 100 | 88.8 | 0 | 75.4 | 59.4 | 0.2 | 47.55 | 10.2 | 6.1 | 88 | 58.04 | 48.52 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=5122026.03 | 99.6 | 99.2 | 0 | 97.2 | 60.6 | 2 | 94.35 | 47.25 | 14.78 | 93.6 | 98.64 | 64.29 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=5122026.03 | 99 | 97.2 | 0 | 84.4 | 40.8 | 0.2 | 70.7 | 30 | 6.2 | 70.27 | 70.92 | 51.79 | |
| H2OLLM=Qwen3-8B, KV Cache Size=10242026.03 | 97.8 | 92 | 2.4 | 81.6 | 38.2 | 2.4 | 72 | 15.85 | 23.98 | 96.2 | 93.92 | 56.03 | |
| DapQLLM=Qwen3-8B, KV Cache Size=1282026.03 | 97.8 | 34 | 0 | 7 | 20.8 | 0 | 0.2 | 0.1 | 1.1 | 40.53 | 5.47 | 18.82 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=2562026.03 | 97.4 | 87.8 | 0 | 81.6 | 35.2 | 0 | 47.2 | 13 | 9.7 | 84.47 | 72.32 | 48.06 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=2562026.03 | 94.8 | 60.4 | 0 | 41.6 | 14.4 | 0 | 5.8 | 5.6 | 1.22 | 34.07 | 24 | 25.63 | |
| H2OLLM=Qwen3-8B, KV Cache Size=5122026.03 | 90.4 | 66.2 | 2.4 | 57.2 | 15 | 0.8 | 30.6 | 5.1 | 14.8 | 89.2 | 65.56 | 39.75 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=1282026.03 | 68.8 | 31.2 | 0 | 2.6 | 3.2 | 0 | 0.15 | 0.45 | 2.06 | 40.33 | 5.24 | 14 | |
| H2OLLM=Qwen3-8B, KV Cache Size=2562026.03 | 68 | 21.4 | 2.4 | 19 | 2.6 | 0 | 6.15 | 1.4 | 11.2 | 61.27 | 19.2 | 19.33 | |
| SLMLLM=Qwen3-8B, KV Cache Size=40962026.03 | 26.6 | 28 | 27 | 24.4 | 19.6 | 18.4 | 26.15 | 27.1 | 36.62 | 93 | 22.96 | 31.8 | |
| H2OLLM=Qwen3-8B, KV Cache Size=1282026.03 | 19.8 | 2.4 | 1.6 | 3.6 | 0.2 | 0 | 0.05 | 0.25 | 8.4 | 22.53 | 3.44 | 5.66 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=40962026.03 | 17.6 | 14.6 | 13.2 | 16.8 | 20.2 | 7.4 | 14.75 | 6.2 | 62.46 | 68.47 | 12.48 | 23.11 | |
| SLMLLM=Qwen3-8B, KV Cache Size=20482026.03 | 13.8 | 13.4 | 11 | 11.4 | 9.4 | 9 | 10.85 | 11.3 | 10.38 | 95.47 | 9.72 | 18.7 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=1282026.03 | 11.4 | 2.8 | 0 | 0 | 1.2 | 0 | 0 | 0 | 0.84 | 4.6 | 3.68 | 2.23 | |
| SLMLLM=Qwen3-8B, KV Cache Size=10242026.03 | 4.6 | 6.2 | 5.6 | 6.6 | 3.8 | 5 | 6.25 | 6.7 | 0.8 | 96.47 | 3.24 | 13.21 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=20482026.03 | 2 | 1.6 | 2.4 | 3.4 | 1.2 | 0 | 2.45 | 0.9 | 43.86 | 69.4 | 2.04 | 11.75 | |
| SLMLLM=Qwen3-8B, KV Cache Size=5122026.03 | 1.8 | 4 | 1.2 | 3.8 | 2.4 | 3 | 3.75 | 4.4 | 0.76 | 97.6 | 0.56 | 11.21 | |
| SLMLLM=Qwen3-8B, KV Cache Size=2562026.03 | 0.8 | 1.2 | 1.2 | 2 | 2.4 | 0 | 2.3 | 2.4 | 0.72 | 95 | 0 | 9.82 | |
| DapQLLM=Qwen3-8B, KV Cache Size=642026.03 | 0.8 | 0.2 | 0 | 0 | 1.2 | 0 | 0 | 0 | 1 | 2.4 | 1.68 | 0.66 | |
| SLMLLM=Qwen3-8B, KV Cache Size=1282026.03 | 0.6 | 1.2 | 1.2 | 2 | 0.2 | 0 | 2.3 | 2.4 | 0.64 | 96 | 0 | 9.69 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=10242026.03 | 0.2 | 1.6 | 2.4 | 3.2 | 0.4 | 0 | 2.45 | 1.05 | 15.86 | 62.07 | 0.32 | 8.14 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=5122026.03 | 0 | 1.6 | 0 | 3 | 0 | 0 | 0 | 0.8 | 7.06 | 26.07 | 0.08 | 3.51 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=2562026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1.7 | 7.93 | 0 | 0.88 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=1282026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.46 | 0.2 | 0 | 0.06 | |
| LaCacheLLM=Qwen3-8B, KV Cache Size=642026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.72 | 0 | 0 | 0.07 | |
| SLMLLM=Qwen3-8B, KV Cache Size=642026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.4 | 32.33 | 0 | 2.98 | |
| H2OLLM=Qwen3-8B, KV Cache Size=642026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3.42 | 0 | 1.24 | 0.42 | |
| PyramidKVLLM=Qwen3-8B, KV Cache Size=642026.03 | 0 | 0 | 0 | 0.2 | 0 | 0 | 0 | 0 | 0.78 | 0.4 | 0.56 | 0.18 | |
| SnapKVLLM=Qwen3-8B, KV Cache Size=642026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.8 | 0.07 | 0.88 | 0.16 |