Long-context evaluation on RULER 128k
98Query Metric (MQ)Llama-3.1-8B
Evaluation Results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8BBackbone=Llama-3.1-8B, KV Cache Budget=Full2025.05 | 98 | 100 | 100 | 100 | 100 | 88 | 64 | 96 | 95.2 | 1.6 | 66.6 | 64 | 36 | 77.6 | — | |
| TailorKV-1Backbone=Llama-3.1-8B, KV Cache Budget=128+(896) tokens2025.05 | 98 | 92 | 92 | 100 | 100 | 64 | 0 | 93 | 67.2 | 0.4 | 16 | 60 | 40 | 63.3 | — | |
| TailorKV-2Backbone=Llama-3.1-8B, KV Cache Budget=128+(896) tokens2025.05 | 97 | 100 | 92 | 100 | 100 | 64 | 16 | 96 | 85.6 | 0.4 | 40 | 64 | 36 | 68.5 | — | |
| Yi-9BBackbone=Yi-9B, KV Cache Budget=Full2025.05 | 84 | 100 | 100 | 100 | 96 | 80 | 28 | 69 | 10.4 | 3.6 | 89.3 | 36 | 36 | 64 | — | |
| TailorKV-2Backbone=Yi-9B, KV Cache Budget=128+(896) tokens2025.05 | 80 | 100 | 100 | 96 | 96 | 76 | 20 | 55 | 48.8 | 24 | 41.3 | 36 | 32 | 61.9 | — | |
| TailorKV-1Backbone=Yi-9B, KV Cache Budget=128+(896) tokens2025.05 | 79.6 | 100 | 100 | 96 | 96 | 72 | 20 | 44 | 19.2 | 23.2 | 44 | 40 | 32 | 58.9 | — | |
| QuestBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 71 | 80 | 68 | 0 | 88 | 48 | 0 | 66 | 59.2 | 0.4 | 52 | 48 | 28 | 46.8 | — | |
| Yi-6BBackbone=Yi-6B, KV Cache Budget=Full2025.05 | 67 | 100 | 100 | 100 | 84 | 72 | 4 | 30 | 4.8 | 1.2 | 100 | 32 | 24 | 55.3 | — | |
| TailorKV-2Backbone=Yi-6B, KV Cache Budget=128+(896) tokens2025.05 | 48 | 100 | 100 | 100 | 84 | 28 | 4 | 21 | 14.4 | 1.2 | 60 | 32 | 20 | 47.1 | — | |
| TailorKV-1Backbone=Yi-6B, KV Cache Budget=128+(896) tokens2025.05 | 47 | 100 | 100 | 100 | 84 | 16 | 0 | 25 | 3.2 | 0.8 | 61.3 | 32 | 22.7 | 45.5 | — | |
| Qwen3-8B-SFT w/ LongRModel=Qwen3, Parameter Size=8B, Training Strategy=SFT w/ LongR2026.02 | 45.59 | — | — | — | — | — | — | — | 80.2 | — | — | — | — | 61.47 | 58.63 | |
| SnapKVBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 38 | 100 | 84 | 0 | 84 | 24 | 0 | 19 | 65.6 | 0 | 28 | 48 | 32 | 40.2 | — | |
| Qwen3-4B-SFT w/ LongRModel=Qwen3, Parameter Size=4B, Training Strategy=SFT w/ LongR2026.02 | 35 | — | — | — | — | — | — | — | 90.84 | — | — | — | — | 60.71 | 56.28 | |
| Qwen3-8B-SFTModel=Qwen3, Parameter Size=8B, Training Strategy=SFT2026.02 | 30.5 | — | — | — | — | — | — | — | 51.96 | — | — | — | — | 48.03 | 61.64 | |
| Qwen3-8B-SFT w/ DAPOModel=Qwen3, Parameter Size=8B, Training Strategy=SFT w/ DAPO2026.02 | 30.4 | — | — | — | — | — | — | — | 76.56 | — | — | — | — | 54.87 | 57.64 | |
| Qwen3-4B-SFT w/ DAPOModel=Qwen3, Parameter Size=4B, Training Strategy=SFT w/ DAPO2026.02 | 28.7 | — | — | — | — | — | — | — | 90.48 | — | — | — | — | 57.92 | 54.57 | |
| QuestBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 28 | 100 | 84 | 4 | 72 | 24 | 0 | 28 | 16.8 | 0.8 | 69.3 | 24 | 32 | 37.1 | — | |
| Qwen3-4B-SFTModel=Qwen3, Parameter Size=4B, Training Strategy=SFT2026.02 | 27.2 | — | — | — | — | — | — | — | 91.44 | — | — | — | — | 57.22 | 53.02 | |
| QuestBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 23 | 96 | 72 | 0 | 72 | 20 | 0 | 17 | 6.4 | 0.8 | 49.3 | 16 | 8 | 29.2 | — | |
| Qwen3-8B-BaseModel=Qwen3, Parameter Size=8B, Training Strategy=Base2026.02 | 17.6 | — | — | — | — | — | — | — | 47.68 | — | — | — | — | 39.05 | 51.88 | |
| Qwen3-4B-BaseModel=Qwen3, Parameter Size=4B, Training Strategy=Base2026.02 | 14 | — | — | — | — | — | — | — | 45.64 | — | — | — | — | 35.46 | 46.74 | |
| PQCacheBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 5 | 8 | 16 | 0 | 24 | 4 | 0 | 2 | 4 | 0.4 | 77.3 | 16 | 28 | 14.2 | — | |
| StreamLLMBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 4 | 0 | 4 | 0 | 0 | 4 | 0 | 5 | 0 | 0.4 | 9.3 | 24 | 20 | 5.4 | — | |
| SnapKVBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 4 | 92 | 12 | 0 | 20 | 4 | 0 | 12 | 7.2 | 2 | 53.3 | 20 | 32 | 19.9 | — | |
| SnapKVBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 4 | 76 | 0 | 0 | 16 | 0 | 0 | 1 | 8 | 0.8 | 69.3 | 20 | 16 | 16.2 | — | |
| PQCacheBackbone=Llama-3.1-8B, KV Cache Budget=1024 tokens2025.05 | 3 | 0 | 8 | 0 | 4 | 8 | 0 | 2 | 0.8 | 0 | 66.6 | 40 | 32 | 12.6 | — | |
| PQCacheBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 2 | 0 | 8 | 0 | 12 | 4 | 0 | 1 | 0 | 0 | 54.6 | 24 | 28 | 10.2 | — | |
| StreamLLMBackbone=Yi-9B, KV Cache Budget=1024 tokens2025.05 | 1.14 | 0 | 4 | 4 | 0 | 0 | 0 | 2 | 0 | 0 | 86.6 | 16 | 24 | 10.6 | — | |
| StreamLLMBackbone=Yi-6B, KV Cache Budget=1024 tokens2025.05 | 1 | 0 | 4 | 4 | 0 | 0 | 0 | 2 | 0 | 0.8 | 68 | 20 | 16 | 8.9 | — |