Long-context language modeling evaluation on RULER
100Single-key AccuracyFull Attention
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full AttentionContext Length=4k2026.03 | 100 | 100 | 100 | 100 | 82.5 | 84.4 | 89.8 | 59.2 | 89.5 | — | — | |
| LycheeClusterContext Length=4k2026.03 | 100 | 100 | 100 | 99.5 | 81.2 | 80.3 | 89.8 | 59.2 | 88.8 | — | — | |
| LycheeClusterContext Length=8k2026.03 | 100 | 100 | 96.4 | 100 | 78 | 83.7 | 81.6 | 59.2 | 87.4 | — | — | |
| Full CacheBackbone=Llama-3.1-8B, Cache Size=100%, Context Length=32K2025.02 | 100 | 100 | 99.75 | 98.25 | 99.4 | 95.33 | 84 | 62 | 91.05 | 44.9 | 0 | |
| SnapKVBackbone=Llama-3.1-8B, Cache Size=40%, Context Length=32K2025.02 | 100 | 93 | 87.5 | 92.75 | 96.6 | 90 | 41 | 52 | 67.93 | 20.3 | 25.4 | |
| SnapKV + CriticalKVBackbone=Llama-3.1-8B, Cache Size=40%, Context Length=32K2025.02 | 100 | 100 | 99 | 99 | 97 | 92.67 | 63 | 53 | 76.89 | 33.9 | 15.6 | |
| AdaKV + CriticalKVBackbone=Llama-3.1-8B, Cache Size=40%, Context Length=32K2025.02 | 100 | 100 | 98.75 | 99.25 | 99.6 | 94.67 | 64 | 58 | 86.28 | 52.4 | 5.2 | |
| HeadKVBackbone=Llama-3.1-8B, Cache Size=40%, Context Length=32K2025.02 | 100 | 99 | 93 | 97 | 98 | 92.67 | 56 | 54 | 79.98 | 17.1 | 12.2 | |
| HeadKV + CriticalKVBackbone=Llama-3.1-8B, Cache Size=40%, Context Length=32K2025.02 | 100 | 100 | 99 | 99.5 | 99.4 | 93.33 | 72 | 59 | 89.29 | 56.5 | 1.9 | |
| Full CacheBackbone=Qwen2.5-32B, Cache Size=100%, Context Length=32K2025.02 | 100 | 99 | 99.25 | 100 | 100 | 96 | 82 | 74 | 93.91 | 90.6 | 0 | |
| SnapKVBackbone=Qwen2.5-32B, Cache Size=40%, Context Length=32K2025.02 | 100 | 63 | 61.75 | 71.75 | 99.8 | 92 | 50 | 64 | 63.86 | 87.9 | 32 | |
| SnapKV + CriticalKVBackbone=Qwen2.5-32B, Cache Size=40%, Context Length=32K2025.02 | 100 | 98 | 99.5 | 99.75 | 100 | 93.33 | 67 | 69 | 81.09 | 88.6 | 13.7 | |
| AdaKVBackbone=Qwen2.5-32B, Cache Size=40%, Context Length=32K2025.02 | 100 | 81 | 83.5 | 87.5 | 100 | 93.33 | 55 | 64 | 71.09 | 88.9 | 24.3 | |
| AdaKV + CriticalKVBackbone=Qwen2.5-32B, Cache Size=40%, Context Length=32K2025.02 | 100 | 98 | 99.5 | 100 | 100 | 94.33 | 64 | 66 | 83.87 | 89.5 | 10.7 | |
| HeadKVBackbone=Qwen2.5-32B, Cache Size=40%, Context Length=32K2025.02 | 100 | 92 | 96.25 | 97.25 | 100 | 94.67 | 68 | 66 | 81.04 | 89.3 | 13.7 | |
| HeadKV + CriticalKVBackbone=Qwen2.5-32B, Cache Size=40%, Context Length=32K2025.02 | 100 | 99 | 98.75 | 100 | 100 | 95.33 | 72 | 69 | 90.69 | 89.9 | 3.4 | |
| AdaKVBackbone=Llama-3.1-8B, Cache Size=40%, Context Length=32K2025.02 | 99 | 98 | 98.25 | 97.5 | 98.6 | 92 | 43 | 54 | 78.38 | 26.6 | 13.9 | |
| Full CacheBackbone=Mistral-7B, Cache Size=100%, Context Length=32K2025.02 | 99 | 94 | 95 | 94.25 | 90.6 | 95.67 | 60 | 59 | 78.27 | 30 | 0 | |
| AdaKV + CriticalKVBackbone=Mistral-7B, Cache Size=40%, Context Length=32K2025.02 | 99 | 89 | 95 | 85 | 89.4 | 95.33 | 60 | 57 | 69.17 | 40.5 | 11.6 | |
| Full AttentionContext Length=32k2026.03 | 95.9 | 95.9 | 97.5 | 97.5 | 76.3 | 76.9 | 79.6 | 59.2 | 84.8 | — | — | |
| LycheeClusterContext Length=32k2026.03 | 95.9 | 98 | 92.9 | 93.9 | 85.7 | 76.2 | 75.5 | 59.2 | 84.7 | — | — | |
| LycheeClusterContext Length=16k2026.03 | 93.9 | 100 | 94.9 | 94.9 | 73.1 | 89.8 | 77.6 | 51 | 84.4 | — | — | |
| Full AttentionContext Length=8k2026.03 | 91.8 | 100 | 100 | 99 | 79.6 | 87.1 | 81.6 | 59.2 | 87.3 | — | — | |
| HeadKV + CriticalKVBackbone=Mistral-7B, Cache Size=40%, Context Length=32K2025.02 | 84 | 53 | 62.25 | 60.5 | 81.4 | 95 | 53 | 55 | 57.59 | 49.5 | 26.4 | |
| SnapKV + CriticalKVBackbone=Mistral-7B, Cache Size=40%, Context Length=32K2025.02 | 74 | 31 | 24.25 | 26 | 81.8 | 94.67 | 45 | 53 | 41.56 | 48.5 | 46.9 | |
| HeadKVBackbone=Mistral-7B, Cache Size=40%, Context Length=32K2025.02 | 70 | 24 | 16.5 | 16.25 | 82.6 | 92.67 | 45 | 53 | 39.59 | 30.6 | 49.4 | |
| SnapKVBackbone=Mistral-7B, Cache Size=40%, Context Length=32K2025.02 | 65 | 16 | 8 | 11.25 | 74.8 | 91.67 | 34 | 51 | 32.15 | 32.2 | 58.9 | |
| Full AttentionContext Length=16k2026.03 | 63.3 | 100 | 98.5 | 98.5 | 82 | 93.9 | 79.6 | 51 | 83.3 | — | — | |
| AdaKVBackbone=Mistral-7B, Cache Size=40%, Context Length=32K2025.02 | 44 | 25 | 14.25 | 20 | 71.6 | 92.33 | 44 | 55 | 34.88 | 25.3 | 55.4 |