Long-context language model evaluation on HELMET
55.2Average ScoreFullKV
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FullKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=Full2026.03 | 55.2 | 74 | 71 | 53.8 | 75.6 | 31.8 | 20.53 | 30.33 | 56 | 49.83 | 57.67 | 86.67 | — | — | — | — | — | — | — | |
| DapQLLM=Qwen2.5-7B-Instruct, KV Cache Size=20482026.03 | 54.78 | 71 | 67.8 | 55 | 76 | 33.6 | 19.05 | 29.46 | 56.33 | 48.83 | 58.5 | 87 | — | — | — | — | — | — | — | |
| SnapKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=20482026.03 | 53.75 | 68.4 | 62 | 51.4 | 78 | 33.2 | 20.35 | 29.41 | 55.67 | 47.83 | 58.33 | 86.67 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=20482026.03 | 53.58 | 68.4 | 61.4 | 53.2 | 77.4 | 33.2 | 19.46 | 28.52 | 53.67 | 48 | 60.83 | 85.33 | — | — | — | — | — | — | — | |
| RecaLLM-Qwen2.5-7B2026.04 | 53.5 | — | — | — | — | — | — | — | — | — | — | — | 96.2 | 69.4 | 69.7 | 13.7 | 46.2 | 35.5 | 44 | |
| RecaLLM-Llama-3.1-8B2026.04 | 53.4 | — | — | — | — | — | — | — | — | — | — | — | 98.2 | 68.1 | 64.1 | 24.7 | 53.2 | 34.8 | 31 | |
| H2OLLM=Qwen2.5-7B-Instruct, KV Cache Size=20482026.03 | 52.24 | 63.2 | 54 | 51 | 79.4 | 31.4 | 20.37 | 28.74 | 52 | 49.33 | 58.17 | 87 | — | — | — | — | — | — | — | |
| DapQLLM=Qwen2.5-7B-Instruct, KV Cache Size=10242026.03 | 52.07 | 66 | 56.4 | 49.8 | 76.2 | 31.2 | 20.2 | 28.76 | 54.33 | 46.67 | 57.5 | 85.67 | — | — | — | — | — | — | — | |
| SnapKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=10242026.03 | 49 | 56.8 | 42.1 | 38 | 75 | 30.8 | 20.53 | 28.05 | 56 | 48 | 58.33 | 85.37 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=10242026.03 | 48.59 | 58.4 | 40.6 | 39 | 78.2 | 29.2 | 20.3 | 28.11 | 50.67 | 44.5 | 61 | 84.5 | — | — | — | — | — | — | — | |
| DapQLLM=Qwen2.5-7B-Instruct, KV Cache Size=5122026.03 | 48.1 | 52 | 43.2 | 40.8 | 72.2 | 28.4 | 20.54 | 29.5 | 54 | 45.33 | 57.33 | 85.83 | — | — | — | — | — | — | — | |
| FullKVLLM=Llama3-8B-Instruct, KV Cache Size=Full2026.03 | 45.75 | 38.6 | 73.6 | 76.2 | 39.4 | 25 | 12 | 16.56 | 52 | 42.17 | 47.67 | 80 | — | — | — | — | — | — | — | |
| H2OLLM=Qwen2.5-7B-Instruct, KV Cache Size=10242026.03 | 45.37 | 44.6 | 33.2 | 28 | 79.4 | 25.4 | 19.52 | 27.24 | 50.67 | 48.5 | 57 | 85.5 | — | — | — | — | — | — | — | |
| SnapKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=5122026.03 | 43.74 | 44.6 | 23.6 | 22.2 | 71.8 | 26.8 | 20.17 | 27.85 | 53 | 47.17 | 58.17 | 85.83 | — | — | — | — | — | — | — | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 42.97 | 26.8 | 64.4 | 72.6 | 39.8 | 15.4 | 11.81 | 16.67 | 50 | 45 | 48.33 | 81.83 | — | — | — | — | — | — | — | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 42.85 | 32 | 64 | 70.6 | 36 | 21.2 | 11.3 | 15.28 | 47.67 | 44.17 | 47.67 | 81.5 | — | — | — | — | — | — | — | |
| DapQLLM=Qwen2.5-7B-Instruct, KV Cache Size=2562026.03 | 42.68 | 41.4 | 25.4 | 27.6 | 67.4 | 21 | 17.64 | 26.55 | 51.33 | 47 | 59 | 85.17 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=5122026.03 | 42.49 | 45 | 20 | 22.8 | 70.6 | 25.8 | 21.91 | 26.6 | 48.67 | 43.5 | 59.67 | 82.83 | — | — | — | — | — | — | — | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 40.89 | 23 | 54.4 | 70.6 | 29 | 18.2 | 11.04 | 17.07 | 52.67 | 43.83 | 48.33 | 81.67 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=10242026.03 | 40.73 | 24.8 | 55.4 | 68.6 | 29.6 | 16 | 11.61 | 16 | 52.33 | 44.17 | 48.17 | 81.33 | — | — | — | — | — | — | — | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 40.58 | 21.6 | 52.8 | 64 | 39.2 | 15.4 | 11.18 | 16.89 | 49 | 46 | 48.17 | 82.17 | — | — | — | — | — | — | — | |
| H2OLLM=Qwen2.5-7B-Instruct, KV Cache Size=5122026.03 | 40.36 | 28.6 | 19.4 | 17 | 76.2 | 17.8 | 18.51 | 26.88 | 51.27 | 45.33 | 57.67 | 85.33 | — | — | — | — | — | — | — | |
| SnapKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=2562026.03 | 39.73 | 38.4 | 15.2 | 18.8 | 60 | 19.8 | 19.9 | 26.55 | 51 | 45.33 | 59.83 | 82.17 | — | — | — | — | — | — | — | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 38.84 | 18 | 51.8 | 63 | 28.8 | 18 | 10.65 | 14.45 | 48.33 | 44.17 | 48 | 82 | — | — | — | — | — | — | — | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 37.47 | 17.6 | 41.4 | 63.6 | 22.2 | 14.2 | 11.34 | 16.67 | 50.67 | 44.5 | 47.17 | 82.83 | — | — | — | — | — | — | — | |
| Qwen2.5-7B-Instruct2026.04 | 37.4 | — | — | — | — | — | — | — | — | — | — | — | 58.8 | 58.6 | 41.7 | 10.3 | 17.7 | 33.2 | 41.3 | |
| H2OLLM=Qwen2.5-7B-Instruct, KV Cache Size=2562026.03 | 37.2 | 21.2 | 12 | 9.2 | 74.4 | 14.4 | 18.26 | 26.12 | 48.67 | 43 | 58.67 | 83.33 | — | — | — | — | — | — | — | |
| ProLong-8B-Instruct-512KContext Length=512K2026.04 | 37 | — | — | — | — | — | — | — | — | — | — | — | 74.3 | 68.7 | 18.6 | 12.4 | 12.5 | 36.4 | 36.5 | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 36.69 | 17 | 37.4 | 47 | 38.4 | 13.8 | 11.66 | 15.7 | 48.67 | 44 | 48.17 | 81.83 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=5122026.03 | 36.38 | 17.4 | 39.8 | 56 | 22.2 | 12.2 | 11.69 | 15.87 | 50.67 | 44.17 | 48 | 82.17 | — | — | — | — | — | — | — | |
| Llama-3.1-8B-Instruct2026.04 | 35.7 | — | — | — | — | — | — | — | — | — | — | — | 77 | 63 | 3 | 10 | 21.3 | 36.3 | 39.2 | |
| PyramidKVLLM=Qwen2.5-7B-Instruct, KV Cache Size=2562026.03 | 35.49 | 34.8 | 14 | 13.2 | 50.8 | 15 | 17.36 | 25.93 | 45 | 40 | 59.17 | 75.17 | — | — | — | — | — | — | — | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 34.23 | 11.6 | 35.4 | 54.2 | 23.4 | 11.2 | 11.21 | 12.82 | 46.67 | 42.33 | 47 | 80.67 | — | — | — | — | — | — | — | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 33.37 | 16.8 | 27.8 | 47.6 | 17.8 | 9 | 11.52 | 14.76 | 50.67 | 42.67 | 46.83 | 81.67 | — | — | — | — | — | — | — | |
| DapQLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 32.6 | 16.4 | 23.4 | 25.6 | 31 | 13.2 | 10.99 | 15.14 | 49.33 | 41.5 | 47.33 | 84.67 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=2562026.03 | 32.26 | 14 | 27.4 | 36.2 | 17 | 10 | 11.36 | 14.94 | 52 | 42.83 | 48 | 81.17 | — | — | — | — | — | — | — | |
| H2OLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 29.16 | 6.8 | 20 | 32.8 | 18 | 6.2 | 10.76 | 12.52 | 45.33 | 41 | 46 | 81.33 | — | — | — | — | — | — | — | |
| SnapKVLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 29 | 13.8 | 19.6 | 21.4 | 14.6 | 8.8 | 10.92 | 13.66 | 46 | 38.83 | 47.07 | 84.33 | — | — | — | — | — | — | — | |
| PyramidKVLLM=Llama3-8B-Instruct, KV Cache Size=1282026.03 | 28.88 | 11.2 | 21.8 | 19.6 | 14.8 | 8.2 | 10.7 | 14.03 | 48 | 39.33 | 46.5 | 83.5 | — | — | — | — | — | — | — | |
| LoongRL-14B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | 63.7 | — | 18.7 | — | — | 49.8 | |
| LoongRL-7B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | 61.6 | — | 11.4 | — | — | 44.5 | |
| QwenLong-L1-32BContext Length=32K2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | 52.9 | — | 14.3 | — | — | 47 |