Long-context understanding on LongBench (Comprehensive Metric Set)
48.37Average ScoreLlama3.1-8B
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama3.1-8BBackbone=Llama3.1-8B, KV cache usage=100.002025.09 | 48.37 | 40.88 | 51.72 | 58.47 | 52.68 | 35.8 | 42.52 | 30.64 | 24.44 | 47.1 | 92.83 | 73 | 0.5 | 39 | 68.56 | 67.48 | — | |
| DuoAttentionBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 48.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaKV w/ oursBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-7BBackbone=Qwen2.5-7B, KV cache usage=100.002025.09 | 47.91 | 33.76 | 51.4 | 57.17 | 56.76 | 29.4 | 38.09 | 31.57 | 24.4 | 46.73 | 92.5 | 72 | 2 | 66.5 | 62.86 | 53.53 | — | |
| HeadKV w/ oursBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 47.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ChunkLLMBackbone=Qwen2.5-7B, KV cache usage=52.102025.09 | 47.67 | 31.09 | 50.52 | 56.95 | 55.61 | 29.41 | 38.27 | 31.25 | 23.16 | 46.53 | 92.5 | 72.5 | 3 | 68 | 62.72 | 53.5 | — | |
| ChunkLLMBackbone=Llama3.1-8B, KV cache usage=53.852025.09 | 47.62 | 39.14 | 49.93 | 53.45 | 52.29 | 34.4 | 42.98 | 31.05 | 23.89 | 47.2 | 93.08 | 72.5 | 0.5 | 38.5 | 68.31 | 66.99 | — | |
| SnapKVBackbone=Llama3.1-8B, KV cache usage=53.752025.09 | 47.32 | 40.17 | 51.16 | 55.73 | 52.83 | 35.52 | 42.78 | 28.21 | 24.31 | 45.6 | 92.88 | 72 | 0.5 | 32.66 | 68.64 | 66.86 | — | |
| SnapKVBackbone=Qwen2.5-7B, KV cache usage=51.622025.09 | 47.21 | 31.68 | 50.56 | 56.26 | 56.11 | 28.19 | 37.64 | 28.95 | 24.51 | 46.44 | 92.5 | 72 | 1.5 | 66 | 62.36 | 53.47 | — | |
| StrmLLMBackbone=Llama3.1-8B, KV cache usage=68.682025.09 | 46.8 | 35.94 | 50.96 | 54.13 | 50.54 | 30.95 | 41.83 | 27.5 | 23.06 | 46.23 | 92.83 | 73 | 0.5 | 39 | 68.43 | 67 | — | |
| AdaKV w/ oursBackbone=Mistral-7B, Cache Budget=40%2025.02 | 46.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StrmLLMBackbone=Qwen2.5-7B, KV cache usage=67.752025.09 | 46.16 | 28.88 | 50.37 | 50.47 | 51.6 | 24.55 | 37.76 | 30.36 | 22.27 | 46.84 | 91.5 | 72 | 2.5 | 67.5 | 62.68 | 53.07 | — | |
| HeadKV w/ oursBackbone=Mistral-7B, Cache Budget=40%2025.02 | 46.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SepLLMBackbone=Llama3.1-8B, KV cache usage=53.082025.09 | 45.95 | 36.23 | 51.16 | 51.81 | 47.7 | 27.83 | 40.85 | 27.12 | 21.84 | 46.89 | 92.83 | 72 | 0.5 | 37.5 | 68.5 | 66.54 | — | |
| SepLLMBackbone=Qwen2.5-7B, KV cache usage=51.852025.09 | 45.14 | 26.43 | 50.18 | 50.29 | 47.25 | 22.83 | 36.34 | 28.31 | 21.14 | 46.71 | 91.5 | 71.5 | 1.5 | 65.5 | 62.88 | 52.84 | — | |
| PyramidKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 44.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HeadKV w/ oursBackbone=Llama-3.1-8B, Cache Budget=20%2025.02 | 43.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaKV w/ oursBackbone=Llama-3.1-8B, Cache Budget=20%2025.02 | 43.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DuoAttentionBackbone=Mistral-7B, Cache Budget=40%2025.02 | 43.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HeadKV w/ oursBackbone=Mistral-7B, Cache Budget=20%2025.02 | 43.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidKVBackbone=Mistral-7B, Cache Budget=40%2025.02 | 43.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaKV w/ oursBackbone=Mistral-7B, Cache Budget=20%2025.02 | 42.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidKVBackbone=Llama-3.1-8B, Cache Budget=20%2025.02 | 40.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidKVBackbone=Mistral-7B, Cache Budget=20%2025.02 | 39.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DuoAttentionBackbone=Llama-3.1-8B, Cache Budget=20%2025.02 | 39.52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PREFILLING-DLLMBase Model=UltraLLaDA, Sparsity=inter + intra-sparsity, Attention Token Budget (B)=5122026.06 | 37.02 | 17.07 | 24.71 | 36.07 | 21.57 | 14.3 | 17.87 | 5.21 | 18.12 | 34.43 | 92.5 | 77.5 | 2.5 | 96.91 | 66.92 | 58.2 | 8.46 | |
| Sparse-dLLMBase Model=UltraLLaDA, Attention Token Budget (B)=5122026.06 | 36.68 | 21.3 | 23.16 | 35.43 | 19.08 | 15.39 | 19.27 | 5.35 | 17.51 | 35.37 | 91.43 | 77.5 | 0.73 | 97.58 | 66.89 | 52.57 | 8.26 | |
| dKV-CacheBase Model=UltraLLaDA2026.06 | 36.29 | 17.86 | 22.36 | 32.99 | 17.89 | 14.25 | 18.39 | 5.41 | 17.18 | 36.3 | 90.53 | 80.5 | 1.48 | 94.88 | 67.96 | 54.61 | 8.04 | |
| VanillaBase Model=UltraLLaDA, Context Window=128K2026.06 | 36 | 17.04 | 22.71 | 33.2 | 15.98 | 12.2 | 16.71 | 5.38 | 17.3 | 37.1 | 90.53 | 79.5 | 1.48 | 96.08 | 67.77 | 54.97 | 8 | |
| Fast-dLLMBase Model=UltraLLaDA2026.06 | 35.98 | 22.03 | 21.58 | 37.31 | 17.31 | 11.04 | 16.97 | 5.3 | 17.71 | 33.18 | 90.53 | 79.5 | 1.36 | 94.47 | 66.08 | 53.1 | 8.2 | |
| PREFILLING-DLLMBase Model=UltraLLaDA, Sparsity=inter-sparsity, Attention Token Budget (B)=5122026.06 | 35.64 | 12.27 | 22.4 | 39.94 | 22.21 | 9.64 | 23.48 | 5.33 | 17.85 | 34.33 | 88.16 | 67 | 2.38 | 90.91 | 65.1 | 62.67 | 8.48 | |
| PREFILLING-DLLMBase Model=Dream-v0-Base-7B, Sparsity=inter + intra-sparsity, Attention Token Budget (B)=5122026.06 | 34.59 | 19.9 | 26.72 | 46.4 | 36.22 | 19.96 | 22.1 | 5.58 | 17.52 | 32.96 | 92.11 | 64.5 | 1.54 | 76.02 | 25.9 | 57.98 | 8.02 | |
| Fast-dLLM v2Base Model=Dream-v0-Base-7B2026.06 | 31.66 | 27.5 | 33.25 | 49.76 | 46 | 22.9 | 35.92 | 4.28 | 17.53 | 31.38 | 78.28 | 43.5 | 4 | 88.5 | 10.41 | 7.12 | 6.2 | |
| DuoAttentionBackbone=Mistral-7B, Cache Budget=20%2025.02 | 31.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaBase Model=Dream-v0-Base-7B, Context Window=2K2026.06 | 28.6 | 20.27 | 19.33 | 40.5 | 29.38 | 15.06 | 20.07 | 5.22 | 17.76 | 33.77 | 81.87 | 60 | 0.5 | 16.5 | 46.95 | 41.99 | 8.42 | |
| PREFILLING-DLLMBase Model=Dream-v0-Base-7B, Sparsity=inter-sparsity, Attention Token Budget (B)=5122026.06 | 22.01 | 12.14 | 18.28 | 38.26 | 15.32 | 9.75 | 16.13 | 5.4 | 17.74 | 29.24 | 70.68 | 46 | 0 | 13.75 | 21.49 | 30.23 | 7.7 | |
| Fast-dLLMBase Model=Dream-v0-Base-7B2026.06 | 20.38 | 9.7 | 18.03 | 36.28 | 11.32 | 7.07 | 11.94 | 5.24 | 13.32 | 31.09 | 70.61 | 46 | 0.27 | 3.5 | 24.62 | 29.23 | 7.88 | |
| YaRNBase Model=Dream-v0-Base-7B, Context Window=128K2026.06 | 17.92 | 7.08 | 16.33 | 32.37 | 9.3 | 6.89 | 10.75 | 5.25 | 14.88 | 29 | 55.04 | 26 | 0.68 | 7.4 | 26.28 | 31.44 | 8.09 | |
| Sparse-dLLMBase Model=Dream-v0-Base-7B, Attention Token Budget (B)=5122026.06 | 8.54 | 3.19 | 9.45 | 20.25 | 7.04 | 4.99 | 6.11 | 5.27 | 5.92 | 13.04 | 10.96 | 0 | 0.23 | 2.62 | 17.01 | 22.46 | 8.13 | |
| dKV-CacheBase Model=Dream-v0-Base-7B2026.06 | 7.32 | 3.57 | 6.6 | 18.33 | 7.02 | 4.44 | 6.34 | 5.32 | 6.07 | 7.71 | 7.17 | 0 | 0.47 | 1.6 | 14.74 | 19.92 | 7.84 |