Long-context language modeling on Ruler llama3-8B-Instruct (test)
100S-NIAH-1FullKV
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FullKVLLM=Llama3-8B-Instruct, KV Cache Size=Full2026.03 | 100 | 100 | 100 | 99.2 | 91.8 | 95.8 | 99.75 | 97.5 | 97.82 | 82.93 | 98.32 | 96.65 | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=20482026.03 | 100 | 90 | 14 | 78 | 47 | 12.6 | 74.7 | 45 | 90.04 | 79.33 | 97.68 | 66.21 | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=20482026.03 | 100 | 100 | 40.8 | 99 | 72.2 | 17.4 | 99 | 96.3 | 83.25 | 67.87 | 98.2 | 79.46 | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=20482026.03 | 100 | 98.4 | 61.4 | 99 | 69 | 19.8 | 98.8 | 94.45 | 90.34 | 73.13 | 97.56 | 81.99 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=20482026.03 | 100 | 99 | 96 | 99.2 | 67.2 | 24.4 | 99.05 | 95.45 | 90.68 | 75 | 97.64 | 85.78 | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 100 | 98.2 | 6.2 | 98.6 | 47.8 | 2 | 97 | 90.8 | 56.14 | 61.53 | 97.08 | 68.67 | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 100 | 96.8 | 15.4 | 98.4 | 44.8 | 3.4 | 96.45 | 87.9 | 65.42 | 64.27 | 96.48 | 69.94 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 100 | 98.4 | 85.8 | 99.2 | 41.8 | 6.2 | 97.4 | 92.5 | 65.74 | 69.33 | 96.76 | 77.56 | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 100 | 95.6 | 0 | 97.4 | 35 | 0.2 | 91.8 | 73.5 | 23.56 | 52.8 | 92.96 | 60.26 | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 100 | 95.6 | — | 96.8 | 30.4 | 0.4 | 91.1 | 71.65 | 30.82 | 53.73 | 94.48 | 60.58 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 100 | 97.8 | 59.6 | 98.6 | 29.8 | 1 | 91.95 | 82.9 | 27.16 | 61.27 | 95.2 | 67.75 | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 100 | 94.8 | 0 | 89.6 | 29.4 | 0 | 73.5 | 35.15 | 9.42 | 39.8 | 75.8 | 49.77 | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 100 | 95 | 0 | 90.2 | 26 | 0 | 76.5 | 36.3 | 13.66 | 45.2 | 91.56 | 52.22 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 100 | 97.6 | 23 | 97.8 | 19.8 | 0 | 79.7 | 55 | 12.8 | 51.87 | 88.04 | 56.87 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 99.6 | 97.6 | 1.4 | 94.4 | 21.4 | 0 | 28.85 | 20.05 | 4.32 | 30.33 | 6.84 | 36.8 | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 99.2 | 91 | 0 | 68.2 | 33.2 | 0 | 26.5 | 9.7 | 2.38 | 25.6 | 18.76 | 34.05 | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 98.8 | 89 | 0 | 60.2 | 35.4 | 0 | 17.25 | 7.45 | 5.18 | 28.53 | 13.24 | 32.28 | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 98.2 | 75.8 | 8 | 62 | 38.8 | 4.8 | 49.05 | 10.6 | 63.4 | 75.07 | 92.2 | 52.54 | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 88.4 | 63.8 | 2.4 | 45 | 25.4 | 1.4 | 24.4 | 2.85 | 46.56 | 64.6 | 69.64 | 39.5 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=642026.03 | 85.2 | 87.4 | 0 | 26.2 | 19.6 | 0 | 3.65 | 1.35 | 0.78 | 0.33 | 3.6 | 20.74 | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 67.2 | 56.8 | 2.4 | 25.4 | 15.6 | 0 | 9.05 | 1.25 | 31.1 | 48.2 | 20.64 | 25.24 | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=642026.03 | 58.8 | 65.4 | 0 | 20.4 | 14 | 0 | 0.75 | 0.4 | 0.14 | 0.07 | 2.28 | 14.75 | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=642026.03 | 48.8 | 47.2 | 0 | 13.4 | 7.2 | 0 | 0.4 | 0.25 | 0.08 | 0 | 0.6 | 10.72 | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 41.4 | 38.8 | 2.4 | 14.8 | 2.8 | 0 | 2.2 | 0.3 | 18.06 | 13 | 7.88 | 12.88 | |
| SLMLLM=Llama3-8B-Instruct, KV Cache Size=20482026.03 | 26.6 | 25.4 | 25.4 | 22.6 | 24.2 | 17.8 | 24.1 | 24.3 | 4.9 | 78.6 | 21.16 | 26.82 | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=642026.03 | 22 | 21.2 | 0 | 3.8 | 0.2 | 0 | 0.4 | 0.25 | 5.7 | 0.07 | 3.52 | 5.19 | |
| LaCacheLLM=Llama3-8B-Instruct, KV Cache Size=20482026.03 | 21 | 27.4 | 1.8 | 29.6 | 29 | 3.2 | 12.2 | 6.45 | 86.08 | 86.07 | 11.04 | 28.53 | |
| SLMLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 13.4 | 13.8 | 11.4 | 11.4 | 13.2 | 10.8 | 10.95 | 11.3 | 0.38 | 75.07 | 8.16 | 16.35 | |
| SLMLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 3.6 | 6.2 | 5.6 | 6.6 | 6.2 | 5.4 | 6.25 | 6.75 | 0.18 | 75.33 | 1.08 | 11.2 | |
| SLMLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 1.2 | 1.2 | 1.2 | 2 | 3.4 | 2.4 | 2.3 | 2.45 | 0.14 | 78.6 | 0 | 8.63 | |
| SLMLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 0.6 | 1.2 | 1.2 | 2 | 2 | 0 | 2.25 | 2.45 | 0.2 | 44.93 | 0 | 5.17 | |
| LaCacheLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 0.2 | 4.8 | 2.4 | 4.2 | 4 | 0 | 2.3 | 2.45 | 55.02 | 86.87 | 1.56 | 14.89 | |
| LaCacheLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 0 | 0.2 | 0 | 2.6 | 0.4 | 0 | 0.05 | 1.4 | 16.52 | 78.8 | 0.08 | 9.1 | |
| LaCacheLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3.64 | 58.2 | 0 | 5.62 | |
| LaCacheLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.58 | 8.4 | 0 | 0.82 | |
| LaCacheLLM=Llama3-8B-Instruct, KV Cache Size=642026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.18 | 0.67 | 0 | 0.08 | |
| SLMLLM=Llama3-8B-Instruct, KV Cache Size=642026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.06 | 27.53 | 0 | 2.51 |