Long-context language understanding on RULER 64k context length
0FWE (Error)Full Attention
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full AttentionAttention mechanism=Full Attention, Budget=Full context, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 0 | 99.4 | 98.4 | 96.8 | 93.7 | 19.8 | 70.4 | 47.6 | 65.8 | — | — | — | — | — | — | — | |
| LycheeDecodeAttention mechanism=Sparse Attention, Budget=4096 tokens, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 0 | 99.6 | 73.2 | 90.3 | 81.7 | 19.9 | 63.4 | 44.4 | 59 | — | — | — | — | — | — | — | |
| E2LLM-CContext Length=64K2024.09 | 6.33 | — | — | — | — | 0.6 | — | — | — | — | 29.5 | 2.63 | 4.9 | — | — | — | |
| E2LLM-RContext Length=64K2024.09 | 10.67 | — | — | — | — | 4.4 | — | — | — | — | 1 | 50.13 | 2.6 | — | — | — | |
| LongLLMLinguaContext Length=64K2024.09 | 13.33 | — | — | — | — | 0.2 | — | — | — | — | 15 | 23.25 | 0.3 | — | — | — | |
| LLoCOContext Length=64K2024.09 | 15.67 | — | — | — | — | 0 | — | — | — | — | 9 | 0.31 | 2.4 | — | — | — | |
| MiniCacheBackbone=Qwen2.5-14B-Instruct-1M, Type=Inter, Compression Rate=1.30, Context Length=64K2025.03 | 18.4 | 35.42 | — | 1.56 | 1.82 | 1.46 | 20.83 | 30.21 | 13.78 | — | — | — | — | 12.5 | 13.54 | 2.08 | |
| E2LLM-C+RContext Length=64K2024.09 | 32.67 | — | — | — | — | 6.2 | — | — | — | — | 34.5 | 9.34 | 0.2 | — | — | — | |
| Training–Inference Consistent Segmented ExecutionBackbone=LLaMA2-7B-32K2026.05 | 34.17 | — | — | — | — | — | — | — | — | — | — | — | 2 | — | — | — | |
| MiniCacheBackbone=Llama-3.1-8B-Instruct, Type=Inter, Compression Rate=1.30, Context Length=64K2025.03 | 58.33 | 89.58 | — | 21.35 | 14.06 | 49.38 | 61.46 | 35.42 | 45.04 | — | — | — | — | 66.67 | 43.75 | 10.42 | |
| Single SVDBackbone=Llama-3.1-8B-Instruct, Type=Intra, Compression Rate=8.40, Context Length=64K2025.03 | 61.11 | 25 | — | 44.79 | 28.91 | 3.54 | 47.92 | 36.46 | 45.71 | — | — | — | — | 51.04 | 61.46 | 96.88 | |
| PyramidKVBackbone=Llama-3.1-8B-Instruct, Type=Intra, Compression Rate=8.00, Context Length=64K2025.03 | 68.06 | 100 | — | 98.44 | 100 | 95.42 | 83.33 | 57.29 | 89.94 | — | — | — | — | 100 | 100 | 96.88 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct, Type=Intra, Compression Rate=8.00, Context Length=64K2025.03 | 68.75 | 100 | — | 97.66 | 100 | 95 | 83.33 | 58.33 | 89.68 | — | — | — | — | 100 | 98.96 | 94.79 | |
| xKVBackbone=Llama-3.1-8B-Instruct, Type=Inter, Compression Rate=8.03, Context Length=64K2025.03 | 78.47 | 100 | — | 96.62 | 96.09 | 86.67 | 78.13 | 56.25 | 88.5 | — | — | — | — | 96.88 | 97.92 | 97.92 | |
| Single SVDBackbone=Qwen2.5-14B-Instruct-1M, Type=Intra, Compression Rate=6.35, Context Length=64K2025.03 | 80.21 | 94.79 | — | 66.15 | 82.81 | 46.04 | 48.96 | 50 | 71.79 | — | — | — | — | 66.67 | 83.33 | 98.96 | |
| KIVI-2Backbone=Llama-3.1-8B-Instruct, Type=Intra, Compression Rate=7.10, Context Length=64K2025.03 | 84.38 | 100 | — | 89.58 | 91.41 | 81.46 | 80.21 | 55.21 | 86.87 | — | — | — | — | 96.88 | 98.96 | 90.63 | |
| Full AttnBackbone=Llama-3.1-8B-Instruct, Type=None, Compression Rate=1.00, Context Length=64K2025.03 | 85.42 | 100 | — | 97.66 | 98.96 | 97.29 | 83.33 | 59.38 | 91.89 | — | — | — | — | 100 | 98.96 | 97.92 | |
| xKV-4Backbone=Qwen2.5-14B-Instruct-1M, Type=Inter, Compression Rate=6.21, Context Length=64K2025.03 | 85.76 | 100 | — | 91.93 | 100 | 95 | 71.88 | 60.42 | 90.19 | — | — | — | — | 97.92 | 100 | 98.96 | |
| PyramidKVBackbone=Qwen2.5-14B-Instruct-1M, Type=Intra, Compression Rate=6.00, Context Length=64K2025.03 | 86.11 | 100 | — | 97.14 | 99.74 | 99.58 | 83.33 | 64.58 | 91.17 | — | — | — | — | 100 | 100 | 81.25 | |
| KIVI-2Backbone=Qwen2.5-14B-Instruct-1M, Type=Intra, Compression Rate=6.40, Context Length=64K2025.03 | 89.58 | 100 | — | 92.97 | 98.18 | 97.71 | 78.13 | 64.58 | 90.45 | — | — | — | — | 98.96 | 95.83 | 88.54 | |
| SnapKVBackbone=Qwen2.5-14B-Instruct-1M, Type=Intra, Compression Rate=6.00, Context Length=64K2025.03 | 90.28 | 100 | — | 95.57 | 99.74 | 99.79 | 81.25 | 65.63 | 91.66 | — | — | — | — | 100 | 100 | 84.38 | |
| Full AttnBackbone=Qwen2.5-14B-Instruct-1M, Type=None, Compression Rate=1.00, Context Length=64K2025.03 | 91.32 | 100 | — | 98.96 | 100 | 99.58 | 80.21 | 64.58 | 93.36 | — | — | — | — | 100 | 100 | 98.96 | |
| CLAAModel=LLaMA-3.1-8B, Context length=64K, Token keep rate=40%2026.02 | — | — | 87.72 | 86.67 | — | — | — | — | 82.01 | 89.85 | 63.8 | — | — | — | — | — | |
| FastKVModel=LLaMA-3.1-8B, Context length=64K, Token keep rate=40%2026.02 | — | — | 87.16 | 86.67 | — | — | — | — | 81.28 | 87.69 | 63.6 | — | — | — | — | — | |
| FullKVModel=LLaMA-3.1-8B, Context length=64K, Token keep rate=40%2026.02 | — | — | 86.88 | 86.67 | — | — | — | — | 83.78 | 98.27 | 63.3 | — | — | — | — | — | |
| GemFilterModel=LLaMA-3.1-8B, Context length=64K, Token keep rate=40%2026.02 | — | — | 58.2 | 88.33 | — | — | — | — | 73.12 | 82.23 | 63.7 | — | — | — | — | — |