Long-context evaluation on RULER 16k
95.02Total ScoreFull-KV
Evaluation Results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full-KVModel=Qwen2.5-32B, Compression Ratio=0%2026.02 | 95.02 | 100 | 100 | 100 | 100 | 99.8 | 100 | 99.95 | 99.85 | 100 | 97.7 | 96.2 | 79.4 | 62.4 | |
| HySparseModel Size=7B2026.02 | 94.1 | 100 | 100 | 100 | 99.4 | 99.6 | 96.6 | 96.7 | 89.8 | 97.2 | 60.8 | 95.5 | — | — | |
| Full-AttnModel Size=80B2026.02 | 93.6 | 100 | 99.8 | 92.6 | 99.6 | 99.2 | 93 | 97.3 | 94.9 | 95.4 | 74.5 | 80.4 | — | — | |
| Full-AttnModel Size=7B2026.02 | 93 | 100 | 100 | 100 | 98.6 | 100 | 96.4 | 99.2 | 99.4 | 94.4 | 37.1 | 97.4 | — | — | |
| Hybrid SWAModel Size=7B2026.02 | 91.6 | 100 | 100 | 99.8 | 97.6 | 99.8 | 96.6 | 97.4 | 94.7 | 99.1 | 23.7 | 98.5 | — | — | |
| HySparseModel Size=80B2026.02 | 90.6 | 100 | 99.8 | 99 | 98.2 | 100 | 99.6 | 92.2 | 91.3 | 90.3 | 40.2 | 86.4 | — | — | |
| LU-KVModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 88.88 | 100 | 100 | 100 | 100 | 86.6 | 46.8 | 99.95 | 99.25 | 100 | 96.02 | 96.2 | 71.6 | 59 | |
| QUOKABackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 88.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full-KVModel=Mistral-7B-v0.3, Compression Ratio=0%2026.02 | 86.63 | 94.2 | 96.4 | 99.6 | 97.4 | 95.6 | 76.8 | 88.65 | 89.5 | 96.28 | 82.22 | 87.93 | 71.6 | 50 | |
| BaselineComp (%)=0, Base Model=LongChat-v1.5-32K2026.06 | 85.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STAR-KVComp (%)=60, Base Model=LongChat-v1.5-32K2026.06 | 84.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 80.81 | 100 | 100 | 100 | 99.8 | 44.2 | 23 | 99.9 | 99 | 99.88 | 95.34 | 99.07 | 44 | 46.4 | |
| LU-KVModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 80.47 | 99.8 | 99.2 | 32 | 84.2 | 71.8 | 78.4 | 85.8 | 84.6 | 99.72 | 95.66 | 93.13 | 65 | 56.8 | |
| LU-KVModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 79.66 | 85.6 | 76.6 | 100 | 87 | 90.8 | 35.2 | 92.85 | 96.45 | 92.16 | 80.78 | 86.8 | 64.6 | 46.8 | |
| UniformModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 74.98 | 100 | 100 | 100 | 100 | 8 | 1 | 99.95 | 99.4 | 98.92 | 90.36 | 99.33 | 36.4 | 41.4 | |
| QUOKABackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 73.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid SWAModel Size=80B2026.02 | 72.7 | 95.2 | 94.8 | 70.8 | 93.2 | 86.4 | 69.4 | 83.2 | 57.8 | 66.7 | 13.3 | 69.2 | — | — | |
| QUOKABackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 72.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUOKABackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 70.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 70.68 | 100 | 100 | 99.8 | 99.6 | 1 | 0.2 | 99.95 | 99.55 | 84.52 | 69.26 | 98.93 | 30.2 | 35.8 | |
| LU-KVModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 69.98 | 70.8 | 78.8 | 18.2 | 83.6 | 79.2 | 67.4 | 76.25 | 67.8 | 95.88 | 78.32 | 84.47 | 62 | 47 | |
| AdaModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 67.51 | 92.6 | 91.2 | 97.4 | 87.8 | 6.8 | 1.2 | 86.45 | 88 | 91.28 | 75.44 | 86.47 | 36.4 | 36.6 | |
| QUOKABackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 66.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniformModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 65.7 | 94.6 | 72.8 | 100 | 78.8 | 7.4 | 0.8 | 86.1 | 94.8 | 94.16 | 65.56 | 90.87 | 32.4 | 35.8 | |
| PyramidModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π2 (KeyDiff Metric)2026.02 | 65.57 | 93.2 | 96.2 | 99.6 | 88.2 | 6.6 | 0.6 | 89.75 | 92 | 94.36 | 36.92 | 88.73 | 31.4 | 34.8 | |
| PaluComp (%)=60, Base Model=LongChat-v1.5-32K2026.06 | 62.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 59.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 59.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 56.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 56.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 54.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 54.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 53.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 52.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 50.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 48.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 48.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 48.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 44.04 | 98.8 | 52.6 | 4.4 | 21.8 | 7 | 4.2 | 18.25 | 14.75 | 99.32 | 88.48 | 94.53 | 29.4 | 39 | |
| UniformModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 43.91 | 97.4 | 55.6 | 3.8 | 25.8 | 4.8 | 2 | 19.6 | 14.4 | 99.28 | 87.14 | 94 | 28 | 39 | |
| KeyDiffBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 41.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 38.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 37.48 | 58 | 38.8 | 2.4 | 20.2 | 12.4 | 5.6 | 16.8 | 12.85 | 92.08 | 71.36 | 86.13 | 33.6 | 37 | |
| PyramidModel=Qwen2.5-32B, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 36.83 | 83.8 | 36 | 2.4 | 19.2 | 2 | 0 | 14.95 | 13.15 | 93.68 | 56.84 | 95.73 | 26.4 | 34.6 | |
| LessIsMoreBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 35.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 32.43 | 50 | 57 | 2.4 | 28 | 4.8 | 0.2 | 21.55 | 16.15 | 62.32 | 31.94 | 82.2 | 32 | 33 | |
| SparQBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 32.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 30.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 30.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniformModel=Mistral-7B-v0.3, Compression Ratio=80%, Eviction Metric=π1 (SnapKV Metric)2026.02 | 29.53 | 40.4 | 16.2 | 2.4 | 14.2 | 6.2 | 1 | 11 | 9.65 | 66.92 | 66.96 | 85.53 | 29.8 | 33.6 | |
| LokiBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 25.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 24.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 24.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 24.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 24.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 20.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 15.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 15.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 12.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 12.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 10.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 9.88 | — | — | — | — | — | — | — | — | — | — | — | — | — |