Long-context understanding on LongBench (Task-Specific Metrics)
62.1Overall Average ScoreDPO w/ LongReward
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| DPO w/ LongRewardBackbone=GLM-4-9B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 62.1 | 71.9 | 58.8 | 48.5 | — | — | — | — | — | |
| DPO w/ ContrastBackbone=GLM-4-9B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 60.5 | 67.8 | 58 | 47.8 | — | — | — | — | — | |
| officially post-trainedBackbone=GLM-4-9B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 60.3 | 67.8 | 56.9 | 47.9 | — | — | — | — | — | |
| DPO w/ LongRewardBackbone=Llama-3.1-8B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 59.9 | 67.8 | 55.8 | 43.2 | — | — | — | — | — | |
| DPO w/ SRMBackbone=GLM-4-9B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 59.9 | 67.5 | 57.4 | 48.2 | — | — | — | — | — | |
| SFTBackbone=GLM-4-9B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 56.6 | 68.4 | 50.9 | 42.1 | — | — | — | — | — | |
| DPO w/ SRMBackbone=Llama-3.1-8B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 56.2 | 65 | 49.6 | 42.7 | — | — | — | — | — | |
| DPO w/ ContrastBackbone=Llama-3.1-8B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 56.2 | 67.8 | 46.2 | 40.3 | — | — | — | — | — | |
| Vanilla / Lossless SDTarget Model=Qwen2.5-72B-Instruct, KV Budget=—2026.06 | 55.81 | 44.23 | 57 | 27.54 | 72.55 | 60 | 66.7 | — | — | |
| DustinTarget Model=Qwen2.5-72B-Instruct, KV Budget=5122026.06 | 55.23 | 44.23 | 56.35 | 27.1 | 71.23 | 60 | 65.97 | — | — | |
| SFTBackbone=Llama-3.1-8B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 55 | 66.1 | 44.5 | 39.6 | — | — | — | — | — | |
| DustinTarget Model=Llama-3.3-70B-Instruct, KV Budget=1282026.06 | 53.81 | 45.55 | 59.02 | 26.75 | 68.18 | 53.96 | 62.78 | — | — | |
| DustinTarget Model=Llama-3.3-70B-Instruct, KV Budget=5122026.06 | 53.23 | 45.27 | 59.1 | 27.76 | 70.14 | 53.4 | 59.15 | — | — | |
| DustinTarget Model=Qwen2.5-72B-Instruct, KV Budget=1282026.06 | 52.41 | 44.59 | 55.31 | 26.14 | 68.35 | 52.13 | 61.3 | — | — | |
| SnapKVTarget Model=Llama-3.3-70B-Instruct, KV Budget=5122026.06 | 52.41 | 44.45 | 59.58 | 24.87 | 68.75 | 53.39 | 58.82 | — | — | |
| SnapKVTarget Model=Qwen2.5-72B-Instruct, KV Budget=5122026.06 | 51.9 | 42.8 | 58.77 | 23.02 | 71.21 | 55 | 57.29 | — | — | |
| Vanilla / Lossless SDTarget Model=Llama-3.3-70B-Instruct, KV Budget=—2026.06 | 50.94 | 45.48 | 58.88 | 28.58 | 70.75 | 53.63 | 49.64 | — | — | |
| ClassicSD / MagicDecTarget Model=Llama-3.1-8B-Instruct, KV Budget=—2026.06 | 50.58 | 42.34 | 44.15 | 29.11 | 69.38 | 52.75 | 59.72 | — | — | |
| DustinTarget Model=Llama-3.1-8B-Instruct, KV Budget=5122026.06 | 50.45 | 42.26 | 44.26 | 28.63 | 69.08 | 52.75 | 59.68 | — | — | |
| SnapKVTarget Model=Llama-3.3-70B-Instruct, KV Budget=1282026.06 | 50.32 | 39.17 | 58.02 | 22.07 | 64.55 | 54.63 | 58.53 | — | — | |
| DeltaKV†Backbone=Llama-3.1-8B, KR=45, CR=30, variant=lightweight decompressor2026.02 | 50.3 | 43.2 | 46.8 | 27.7 | 69.5 | 54.8 | 59.7 | 45 | 30 | |
| DeltaKVBackbone=Llama-3.1-8B, KR=29, CR=30, quantization=4-bit2026.02 | 50.3 | 43.3 | 46.6 | 27.3 | 69.8 | 54.4 | 60.5 | 29 | 30 | |
| OmniKVBackbone=Llama-3.1-8B, KR=100, CR=302026.02 | 50.2 | 44.8 | 46.1 | 28.9 | 68.9 | 52.8 | 59.9 | 100 | 30 | |
| DeltaKVBackbone=Llama-3.1-8B, KR=45, CR=302026.02 | 50.2 | 44.4 | 45.9 | 27.6 | 69.7 | 53.4 | 60.2 | 45 | 30 | |
| OmniKVBackbone=Llama-3.1-8B, KR=100, CR=202026.02 | 50.1 | 43 | 45.7 | 27.5 | 68.6 | 54.9 | 60.8 | 100 | 20 | |
| GPT-4Length=128k2024.08 | 50 | — | — | — | — | — | — | — | — | |
| Llama-3.1-8BBackbone=Llama-3.1-8B, KR=100, CR=1002026.02 | 50 | 45.3 | 46.2 | 28.7 | 69.4 | 52.7 | 57.9 | 100 | 100 | |
| QuestBackbone=Llama-3.1-8B, KR=100, CR=302026.02 | 50 | 44.4 | 46.2 | 29.3 | 69 | 53.1 | 57.6 | 100 | 30 | |
| DustinTarget Model=Llama-3.1-8B-Instruct, KV Budget=2562026.06 | 49.81 | 42.15 | 44.31 | 27.72 | 68.77 | 52.59 | 58.11 | — | — | |
| SnapKVBackbone=Llama-3.1-8B, KR=30, CR=302026.02 | 49.8 | 44.5 | 46.4 | 26.5 | 68.2 | 52.7 | 60.3 | 30 | 30 | |
| DeltaKV†Backbone=Llama-3.1-8B, KR=43, CR=20, variant=lightweight decompressor2026.02 | 49.8 | 42.7 | 46.2 | 26.2 | 68.7 | 54.3 | 60.5 | 43 | 20 | |
| AdaKVBackbone=Llama-3.1-8B, KR=30, CR=302026.02 | 49.7 | 44.7 | 46.5 | 26.6 | 69.3 | 52.7 | 58.3 | 30 | 30 | |
| QuestBackbone=Llama-3.1-8B, KR=100, CR=202026.02 | 49.7 | 42.2 | 46.9 | 28.8 | 68.9 | 54.9 | 56.5 | 100 | 20 | |
| PyramidKVBackbone=Llama-3.1-8B, KR=30, CR=302026.02 | 49.5 | 44.1 | 46.3 | 26 | 68.5 | 52.6 | 59.1 | 30 | 30 | |
| officially post-trainedBackbone=Llama-3.1-8B, Judge Model=GPT-4o, Prompting Protocol=zero-shot2024.10 | 49.4 | 59.3 | 42.9 | 35.3 | — | — | — | — | — | |
| SnapKVBackbone=Llama-3.1-8B, KR=20, CR=202026.02 | 49.2 | 42.9 | 45.4 | 25.8 | 68.8 | 54.8 | 57.4 | 20 | 20 | |
| DustinTarget Model=Llama-3.1-8B-Instruct, KV Budget=1282026.06 | 49.02 | 41.95 | 43.99 | 25.03 | 68.42 | 52.84 | 57.15 | — | — | |
| Quest*Target Model=Llama-3.3-70B-Instruct, KV Budget=5122026.06 | 48.96 | 40.34 | 52.24 | 27.18 | 66.63 | 54.61 | 51.95 | — | — | |
| Gemini-1.5-ProLength=128k2024.08 | 48.5 | — | — | — | — | — | — | — | — | |
| ChatGLM3-6BModel Scale=< 7B2024.03 | 48.4 | 40.9 | 45 | 26 | 56.5 | 65 | 57.1 | — | — | |
| Qwen2.5-32BBackbone=Qwen2.5-32B, KR=100, CR=1002026.02 | 48.4 | 42.7 | 54.3 | 27.3 | 67.6 | 56 | 42.6 | 100 | 100 | |
| InternLM2-Chat-7B-SFTModel Scale=< 7B2024.03 | 48.1 | 47.3 | 45.2 | 25.3 | 59.9 | 67.2 | 43.5 | — | — | |
| OmniKVBackbone=Qwen2.5-32B, KR=100, CR=202026.02 | 48.1 | 42.4 | 54.2 | 26.9 | 67.2 | 56.1 | 41.7 | 100 | 20 | |
| DeltaKVBackbone=Qwen2.5-32B, KR=44, CR=202026.02 | 47.7 | 41.9 | 54.2 | 26.2 | 66.1 | 55.4 | 42.1 | 44 | 20 | |
| Quest*Target Model=Qwen2.5-72B-Instruct, KV Budget=5122026.06 | 47.69 | 36.91 | 41.8 | 25.92 | 64.4 | 55.08 | 57.24 | — | — | |
| Qwen2.5-7BBackbone=Qwen2.5-7B, KR=100, CR=1002026.02 | 47.6 | 42.5 | 49.6 | 28.6 | 68.7 | 53.8 | 42.5 | 100 | 100 | |
| OmniKVBackbone=Qwen2.5-7B, KR=100, CR=302026.02 | 47.4 | 41.9 | 49.4 | 28.4 | 69.1 | 54 | 41.5 | 100 | 30 | |
| DeltaKVBackbone=Qwen2.5-7B, KR=48, CR=302026.02 | 47.1 | 41.8 | 49 | 27.7 | 69.1 | 53.3 | 41.7 | 48 | 30 | |
| SnapKVBackbone=Qwen2.5-32B, KR=20, CR=202026.02 | 47.1 | 39.5 | 53.8 | 24.6 | 67.2 | 56 | 41.7 | 20 | 20 | |
| SnapKVTarget Model=Qwen2.5-72B-Instruct, KV Budget=1282026.06 | 47.08 | 36.57 | 56.89 | 19.8 | 63.71 | 51.63 | 51.54 | — | — | |
| SnapKVBackbone=Qwen2.5-7B, KR=30, CR=302026.02 | 47 | 41.7 | 48.8 | 26.6 | 68.3 | 54.5 | 41.8 | 30 | 30 | |
| InternLM2-Chat-20B-SFTModel Scale=13~20B2024.03 | 46.8 | 46.8 | 48.7 | 25.6 | 51.2 | 67.9 | 40.4 | — | — | |
| InternLM2-Chat-7BModel Scale=< 7B2024.03 | 46.1 | 45.7 | 43.1 | 26.5 | 58.3 | 66.3 | 36.4 | — | — | |
| PyramidKVBackbone=Qwen2.5-7B, KR=30, CR=302026.02 | 46.1 | 40.6 | 48.7 | 24.4 | 67.7 | 54.5 | 40.6 | 30 | 30 | |
| InternLM2-Chat-20BModel Scale=13~20B2024.03 | 44.8 | 46.9 | 46.7 | 26 | 49.1 | 67.3 | 32.6 | — | — | |
| QuestTarget Model=Llama-3.1-8B-Instruct, KV Budget=5122026.06 | 43.9 | 33.6 | 34.7 | 27.21 | 55.94 | 51.33 | 54.92 | — | — | |
| Original Promptcompression rate=none, LLM=LLaMA 3.1-8B2025.07 | 43.03 | 37.35 | 36.36 | 27.39 | 71.01 | — | — | — | — | |
| Llama3.1-70B-InstructLength=128k2024.08 | 42.8 | — | — | — | — | — | — | — | — | |
| FullBackbone=Llama-3.1-8B2026.02 | 41.47 | 47.51 | 43.28 | 25.9 | 45.92 | 68.18 | 18.01 | — | — | |
| MInferenceBackbone=Llama-3.1-8B2026.02 | 41.14 | 47.42 | 42.54 | 25.85 | 45.58 | 67.6 | 17.84 | — | — | |
| PrismBackbone=Llama-3.1-8B2026.02 | 41.08 | 47.09 | 42.13 | 26 | 46.4 | 66.15 | 18.72 | — | — | |
| XAttentionBackbone=Llama-3.1-8B2026.02 | 39.68 | 45.89 | 41.56 | 26.18 | 45.86 | 59.32 | 19.24 | — | — | |
| StreamingLLMTarget Model=Llama-3.3-70B-Instruct, KV Budget=5122026.06 | 39.61 | 32.36 | 39.35 | 23.01 | 51.55 | 51.05 | 41.98 | — | — | |
| FullBackbone=Qwen-3-8B2026.02 | 39.49 | 47.1 | 40.45 | 24.07 | 56.69 | 67 | 1.65 | — | — | |
| MInferenceBackbone=Qwen-3-8B2026.02 | 39.18 | 46.9 | 40.39 | 24.07 | 55.74 | 66.33 | 1.61 | — | — | |
| PrismBackbone=Qwen-3-8B2026.02 | 39.12 | 46.47 | 40.08 | 24.01 | 58.36 | 64.17 | 1.64 | — | — | |
| XAttentionBackbone=Qwen-3-8B2026.02 | 38.82 | 44.49 | 40.09 | 24.12 | 57.27 | 65.67 | 1.29 | — | — | |
| PaluBackbone=Qwen2.5-7B, KR=50, CR=1002026.02 | 38.8 | 34.4 | 35.6 | 27.5 | 68.7 | 45 | 21.3 | 50 | 100 | |
| Mistral-7B-Instruct-v0.2Model Scale=< 7B2024.03 | 38.3 | 31.3 | 26.4 | 21.8 | 46.6 | 59.2 | 44.8 | — | — | |
| QuestTarget Model=Llama-3.1-8B-Instruct, KV Budget=2562026.06 | 38.15 | 27.61 | 24.64 | 24.45 | 48.67 | 48.37 | 49.86 | — | — | |
| Quest*Target Model=Qwen2.5-72B-Instruct, KV Budget=1282026.06 | 37.01 | 23.12 | 25.21 | 21.67 | 52.81 | 49.7 | 46.37 | — | — | |
| Mixtral-8x7B-Instruct-v0.1Model Scale=13~20B2024.03 | 37 | 35 | 25.6 | 17.1 | 35.9 | 68.2 | 40 | — | — | |
| StreamingLLMTarget Model=Llama-3.3-70B-Instruct, KV Budget=1282026.06 | 36.94 | 27.13 | 39.78 | 19.02 | 46.43 | 52.44 | 39.49 | — | — | |
| FlexPrefillBackbone=Qwen-3-8B2026.02 | 36.13 | 43.77 | 39.31 | 23.99 | 57.33 | 50.5 | 1.87 | — | — | |
| StreamingLLMTarget Model=Llama-3.1-8B-Instruct, KV Budget=5122026.06 | 36 | 28.35 | 30.11 | 23.16 | 50.51 | 47.11 | 38.31 | — | — | |
| CREAM-7B-32kContext Window Length=32k, Instruction-tuning steps=100, Training data length=4K, base_model=LLaMa2-7B-Chat2024.06 | 35.9 | 34.8 | 31.1 | 27.2 | 65.1 | 7 | 50.4 | — | — | |
| DACcompression rate=0.5, SLM=LLaMA 3.2-1B, LLM=LLaMA 3.1-8B2025.07 | 35.42 | 32.68 | 28.08 | 22.92 | 58.01 | — | — | — | — | |
| StreamingLLMTarget Model=Qwen2.5-72B-Instruct, KV Budget=5122026.06 | 35.25 | 27.61 | 33.82 | 21.7 | 54.57 | 39.84 | 35.02 | — | — | |
| LLMLingua-2compression rate=0.5, SLM=LLaMA 3.2-1B, LLM=LLaMA 3.1-8B2025.07 | 34.82 | 32.51 | 27.93 | 22.74 | 56.08 | — | — | — | — | |
| StreamingLLMTarget Model=Llama-3.1-8B-Instruct, KV Budget=2562026.06 | 34.39 | 25.84 | 29.73 | 21.16 | 47.6 | 47.06 | 36.83 | — | — | |
| LongChat-v1.5-7B-32kContext Window Length=32k2024.06 | 34.3 | 28.7 | 20.6 | 26.7 | 60 | 15.8 | 54.1 | — | — | |
| FlexPrefillBackbone=Llama-3.1-8B2026.02 | 33.9 | 46.13 | 41.49 | 25.85 | 46.63 | 25.61 | 17.68 | — | — | |
| LLMLinguacompression rate=0.5, SLM=LLaMA 3.2-1B, LLM=LLaMA 3.1-8B2025.07 | 33.31 | 27.81 | 24.68 | 23.13 | 57.63 | — | — | — | — | |
| Quest*Target Model=Llama-3.3-70B-Instruct, KV Budget=1282026.06 | 33.21 | 23.66 | 27.93 | 21.09 | 47.94 | 44.2 | 35.67 | — | — | |
| StreamingLLMTarget Model=Llama-3.1-8B-Instruct, KV Budget=1282026.06 | 32.86 | 25.02 | 29.46 | 19.25 | 43.94 | 45.23 | 35.79 | — | — | |
| StreamingLLMTarget Model=Qwen2.5-72B-Instruct, KV Budget=1282026.06 | 32.72 | 23.27 | 32.49 | 17.58 | 48.48 | 42.69 | 33.68 | — | — | |
| Selective-Contextcompression rate=0.5, SLM=LLaMA 3.2-1B, LLM=LLaMA 3.1-8B2025.07 | 32.16 | 26.4 | 24.78 | 21.98 | 55.47 | — | — | — | — | |
| Qwen2-7B-InstructLength=128k2024.08 | 31.9 | — | — | — | — | — | — | — | — | |
| Vicuna-v1.5-7B-16kContext Window Length=16k2024.06 | 31.9 | 28 | 18.6 | 26 | 66.2 | 5.5 | 47.3 | — | — | |
| Gradient-Llama3-70BLength=128k2024.08 | 31.8 | — | — | — | — | — | — | — | — | |
| WINABackbone=Phi-4-16k, Sparsity=0.52025.05 | 31.39 | 21.19 | 15.98 | 13.39 | 59.39 | 48.76 | 36.53 | — | — | |
| Qwen2-72B-InstructLength=128k2024.08 | 31 | — | — | — | — | — | — | — | — | |
| Llama2-7B-chat-4kContext Window Length=4k2024.06 | 31 | 24.9 | 22.6 | 24.7 | 60 | 5.9 | 48.1 | — | — | |
| Baichuan2-13B-ChatModel Scale=13~20B2024.03 | 30.9 | 34.3 | 29.1 | 21.3 | 45.5 | 4 | 51.4 | — | — | |
| TEALBackbone=Phi-4-16k, Sparsity=0.52025.05 | 30.54 | 22.15 | 11.69 | 15.04 | 58.44 | 51.03 | 29.51 | — | — | |
| Yi-9B-200kLength=128k2024.08 | 30.3 | — | — | — | — | — | — | — | — | |
| WINABackbone=Phi-4-16k, Sparsity=0.652025.05 | 30.26 | 16.3 | 12.24 | 17.36 | 59.22 | 46.58 | 37.6 | — | — | |
| TEALBackbone=Phi-4-16k, Sparsity=0.652025.05 | 30.07 | 19.25 | 14.11 | 19.5 | 59.48 | 43.52 | 25.71 | — | — | |
| Llama3.1-8B-InstructLength=128k2024.08 | 30 | — | — | — | — | — | — | — | — | |
| Qwen-14B-ChatModel Scale=13~20B2024.03 | 29.7 | 32.6 | 19.4 | 22.5 | 36.9 | 23.7 | 42.9 | — | — |