Long-context evaluation on RULER 64k
100VT ScoreLlama-3.1-8B
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8BBackbone=Llama-3.1-8B, KV Cache Budget=Full2025.05 | 100 | 100 | 100 | 100 | 100 | 100 | 96 | 99 | 100 | 14.8 | 92 | 60 | 52 | 85.6 | — | — | — | — | — | — | — | |
| DenseBackbone=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=1.00, Context Length=64K2025.03 | 97.29 | 100 | 100 | — | 98.96 | 97.92 | — | 97.66 | 98.96 | — | 85.42 | 83.33 | 59.38 | 91.89 | — | — | — | — | — | — | — | |
| Full Attn2026.05 | 96.8 | — | — | — | — | — | — | — | — | 65.28 | 84 | — | — | 86.23 | 100 | — | 63.4 | 69.83 | 99.5 | 97.6 | 99.66 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.00, Context Length=64K2025.03 | 95 | 100 | 100 | — | 98.96 | 94.79 | — | 97.66 | 100 | — | 68.75 | 83.33 | 58.33 | 89.68 | — | — | — | — | — | — | — | |
| Quest2026.05 | 94.8 | — | — | — | — | — | — | — | — | 36.56 | 63.8 | — | — | 70.6 | 87.93 | — | 54.6 | 62.57 | 80.4 | 75.9 | 78.86 | |
| RTPurboselection_mode=top-p2026.05 | 94.6 | — | — | — | — | — | — | — | — | 65.1 | 81.4 | — | — | 85.49 | 99.93 | — | 62.6 | 70.03 | 99.65 | 97.5 | 98.6 | |
| Qwen3-4B-SFT w/ LongRModel=Qwen3, Parameter Size=4B, Training Strategy=SFT w/ LongR2026.02 | 93.32 | — | — | — | — | — | — | — | — | — | — | — | — | 69.43 | 71.36 | 43.6 | — | — | — | — | — | |
| FlexPrefill2026.05 | 92.72 | — | — | — | — | — | — | — | — | 51.04 | 80.07 | — | — | 77.77 | 100 | — | 58 | 68.63 | 96.35 | 55.5 | 97.6 | |
| Qwen3-4B-SFT w/ DAPOModel=Qwen3, Parameter Size=4B, Training Strategy=SFT w/ DAPO2026.02 | 91.96 | — | — | — | — | — | — | — | — | — | — | — | — | 63.53 | 67.74 | 30.9 | — | — | — | — | — | |
| RazorAttn2026.05 | 91.52 | — | — | — | — | — | — | — | — | 65.96 | 83.8 | — | — | 85.11 | 99.73 | — | 62.8 | 69.33 | 98.95 | 95.2 | 98.73 | |
| TokenButlerBackbone=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.00, Context Length=64K2025.03 | 91.46 | 100 | 100 | — | 100 | 96.88 | — | 94.27 | 98.96 | — | 77.08 | 83.33 | 57.29 | 89.93 | — | — | — | — | — | — | — | |
| Qwen3-4B-SFTModel=Qwen3, Parameter Size=4B, Training Strategy=SFT2026.02 | 90.92 | — | — | — | — | — | — | — | — | — | — | — | — | 59.11 | 66.2 | 20.2 | — | — | — | — | — | |
| SnapKV2026.05 | 90 | — | — | — | — | — | — | — | — | 60.08 | 72 | — | — | 75.81 | 98.73 | — | 47.7 | 68.15 | 91.15 | 60.5 | 94 | |
| TailorKV-2Backbone=Llama-3.1-8B, KV Cache Budget=128+(896) tokens2025.05 | 88 | 100 | 100 | 100 | 100 | 96 | 68 | 97 | 98 | 19.6 | 62.7 | 60 | 56 | 80.4 | — | — | — | — | — | — | — | |
| xKVBackbone=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.03, Context Length=64K2025.03 | 86.67 | 100 | 96.88 | — | 97.92 | 97.92 | — | 96.62 | 96.09 | — | 78.47 | 78.13 | 56.25 | 88.5 | — | — | — | — | — | — | — | |
| Qwen3-8B-SFT w/ LongRModel=Qwen3, Parameter Size=8B, Training Strategy=SFT w/ LongR2026.02 | 86.64 | — | — | — | — | — | — | — | — | — | — | — | — | 71.98 | 69.88 | 59.41 | — | — | — | — | — | |
| TailorKV-1Backbone=Llama-3.1-8B, KV Cache Budget=128+(896) tokens2025.05 | 85.6 | 100 | 100 | 96 | 100 | 96 | 28 | 99 | 100 | 18.4 | 57.3 | 56 | 48 | 75.7 | — | — | — | — | — | — | — | |
| Minference2026.05 | 83.48 | — | — | — | — | — | — | — | — | 61.84 | 82.73 | — | — | 65.61 | 86.53 | — | 42.8 | 29.67 | 82.25 | 81.1 | 40.07 | |
| QuestBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 80 | 88 | 100 | 60 | 92 | 72 | 0 | 93 | 90 | 8.4 | 70.6 | 52 | 52 | 66 | — | — | — | — | — | — | — | |
| QuestBackbone=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.00, Context Length=64K2025.03 | 77.71 | 90.75 | 90.63 | — | 96.88 | 87.5 | — | 85.42 | 94.27 | — | 81.94 | 83.33 | 57.29 | 84.57 | — | — | — | — | — | — | — | |
| SnapKVBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 74.4 | 96 | 84 | 0 | 88 | 32 | 0 | 40 | 69 | 0.8 | 41.3 | 56 | 48 | 48.4 | — | — | — | — | — | — | — | |
| KVzipBackbone=Llama-3.1-8B-Instruct, K.V. BW (x ↓)=8.00, Context Length=64K2025.03 | 73.96 | 100 | 100 | — | 8.33 | 70.83 | — | 78.12 | 52.6 | — | 75.35 | 62.5 | 58.33 | 68 | — | — | — | — | — | — | — | |
| RTPurboselection_mode=top-k2026.05 | 69.6 | — | — | — | — | — | — | — | — | 59.92 | 62 | — | — | 70.53 | 76.53 | — | 56 | 64.27 | 98.15 | 97.6 | 50.66 | |
| Qwen3-8B-SFT w/ DAPOModel=Qwen3, Parameter Size=8B, Training Strategy=SFT w/ DAPO2026.02 | 58.52 | — | — | — | — | — | — | — | — | — | — | — | — | 55.02 | 66.13 | 40.4 | — | — | — | — | — | |
| Qwen3-4B-BaseModel=Qwen3, Parameter Size=4B, Training Strategy=Base2026.02 | 56.4 | — | — | — | — | — | — | — | — | — | — | — | — | 45.42 | 60.55 | 19.3 | — | — | — | — | — | |
| PQCacheBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 52.8 | 36 | 60 | 12 | 68 | 48 | 4 | 16 | 37 | 0 | 73.3 | 56 | 48 | 39.2 | — | — | — | — | — | — | — | |
| TailorKV-2Backbone=Yi-9B, KV Cache Budget=128+(896) tokens2025.05 | 42.4 | 100 | 100 | 92 | 100 | 84 | 28 | 62 | 90 | 33.6 | 48 | 28 | 48 | 65.8 | — | — | — | — | — | — | — | |
| TailorKV-2Backbone=Yi-6B, KV Cache Budget=128+(896) tokens2025.05 | 42.4 | 100 | 100 | 100 | 100 | 24 | 28 | 40 | 67 | 0.8 | 57.3 | 32 | 24 | 55.1 | — | — | — | — | — | — | — | |
| PQCacheBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 31.2 | 32 | 56 | 4 | 36 | 16 | 0 | 7 | 39 | 6 | 66.6 | 20 | 36 | 26.9 | — | — | — | — | — | — | — | |
| Qwen3-8B-SFTModel=Qwen3, Parameter Size=8B, Training Strategy=SFT2026.02 | 31 | — | — | — | — | — | — | — | — | — | — | — | — | 44.25 | 62.96 | 38.8 | — | — | — | — | — | |
| TailorKV-1Backbone=Yi-6B, KV Cache Budget=128+(896) tokens2025.05 | 28.8 | 100 | 100 | 100 | 96 | 12 | 24 | 41 | 65 | 1.2 | 58.7 | 28 | 24 | 52.2 | — | — | — | — | — | — | — | |
| Qwen3-8B-BaseModel=Qwen3, Parameter Size=8B, Training Strategy=Base2026.02 | 28.08 | — | — | — | — | — | — | — | — | — | — | — | — | 43.18 | 61.76 | 39.7 | — | — | — | — | — | |
| Yi-6BBackbone=Yi-6B, KV Cache Budget=Full2025.05 | 24.8 | 100 | 100 | 100 | 96 | 56 | 24 | 39 | 76 | 0.8 | 73.3 | 32 | 24 | 57.3 | — | — | — | — | — | — | — | |
| SnapKVBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 22.4 | 80 | 28 | 0 | 20 | 4 | 0 | 11 | 11 | 5.6 | 48 | 24 | 44 | 22.9 | — | — | — | — | — | — | — | |
| QuestBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 20 | 72 | 84 | 0 | 52 | 20 | 0 | 28 | 30 | 1.6 | 56 | 24 | 20 | 31.3 | — | — | — | — | — | — | — | |
| QuestBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 16 | 68 | 92 | 20 | 68 | 40 | 0 | 24 | 42 | 10.8 | 62.6 | 28 | 36 | 39 | — | — | — | — | — | — | — | |
| SnapKVBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 15.2 | 88 | 4 | 0 | 16 | 0 | 0 | 5 | 7 | 0 | 65.3 | 28 | 20 | 19.1 | — | — | — | — | — | — | — | |
| Yi-9BBackbone=Yi-9B, KV Cache Budget=Full2025.05 | 12.8 | 100 | 100 | 100 | 100 | 92 | 48 | 61 | 88 | 15.6 | 88 | 32 | 48 | 68.1 | — | — | — | — | — | — | — | |
| PQCacheBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 10.4 | 16 | 20 | 0 | 28 | 8 | 0 | 5 | 3 | 0 | 50.6 | 24 | 24 | 14.5 | — | — | — | — | — | — | — | |
| TailorKV-1Backbone=Yi-9B, KV Cache Budget=128+(896) tokens2025.05 | 6.4 | 100 | 100 | 92 | 100 | 84 | 28 | 53 | 89 | 32 | 49.3 | 32 | 48 | 62.6 | — | — | — | — | — | — | — | |
| StreamLLMBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 2.4 | 8 | 4 | 0 | 8 | 0 | 0 | 5 | 3 | 0.8 | 72 | 28 | 28 | 12.2 | — | — | — | — | — | — | — | |
| StreamLLMBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 0 | 0 | 4 | 0 | 4 | 0 | 0 | 1 | 0 | 1.2 | 74.6 | 16 | 28 | 9.9 | — | — | — | — | — | — | — | |
| StreamLLMBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 3 | 0 | 0.4 | 62.6 | 20 | 16 | 8.5 | — | — | — | — | — | — | — |