Long-context language modeling on InfiniteBench (Accuracy and Sparsity)
46.19Code Debug AccuracyFast-dLLM v2
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fast-dLLM v2Base Model=Dream-v0-Base-7B2026.06 | 46.19 | — | — | — | — | — | — | — | — | — | — | — | 24.29 | — | 27.12 | — | 27.12 | — | 30.32 | — | 0 | 57.21 | |
| S2OModel backbone=Llama-3.1-8B2026.02 | 24.4 | 17 | 91.78 | 16.6 | 92.2 | 27.1 | 92.25 | 12 | 91.5 | 11 | 91.06 | 90.96 | 34.3 | 91.54 | 98.6 | 93.9 | 99.8 | 93.82 | 37.9 | 92.11 | — | — | |
| FullModel backbone=Llama-3.1-8B2026.02 | 24.1 | 16.7 | 0 | 18.7 | 0 | 27.1 | 0 | 13.5 | 0 | 13.2 | 0 | 0 | 33.1 | 0 | 99.3 | 0 | 99.3 | 0 | 38.3 | 0 | — | — | |
| XAttnModel backbone=Llama-3.1-8B2026.02 | 23.4 | 18 | 71.06 | 16.5 | 71.32 | 22.3 | 71.28 | 13.5 | 72.47 | 8 | 68.31 | 73.03 | 32.9 | 70.75 | 99.8 | 68.92 | 99.5 | 68.56 | 37.1 | 70.63 | — | — | |
| PBSModel backbone=Llama-3.1-8B2026.02 | 22.8 | 17.9 | 75.39 | 19.2 | 75.78 | 24.5 | 75.7 | 11.5 | 77.09 | 12.7 | 72.92 | 77.66 | 32.9 | 75.13 | 83.1 | 74.15 | 99.2 | 73.76 | 36 | 75.29 | — | — | |
| FlexPrefillModel backbone=Llama-3.1-8B2026.02 | 21.8 | 16.7 | 85.91 | 17.8 | 85.23 | 24.5 | 85.34 | 12.5 | 84.16 | 11.3 | 84.8 | 83.76 | 32.8 | 86.82 | 100 | 87.6 | 98.8 | 85.1 | 37.4 | 85.41 | — | — | |
| XAttnModel backbone=Qwen3-8B2026.02 | 20 | 16.4 | 66.31 | 1.9 | 67.01 | 41.5 | 66.97 | 16.5 | 69.14 | 3.7 | 66.69 | 71.47 | 30.6 | 61.89 | 100 | 62.29 | 81 | 62.12 | 34.6 | 65.99 | — | — | |
| S2OModel backbone=Qwen3-8B2026.02 | 20 | 17.5 | 90.85 | 2 | 90.89 | 44.5 | 89.8 | 28 | 86.69 | 3.8 | 86.63 | 87.45 | 32.3 | 87.36 | 100 | 85.54 | 99.7 | 90.33 | 38.6 | 88.39 | — | — | |
| FullModel backbone=Qwen3-8B2026.02 | 18.5 | 18 | 0 | 2.2 | 0 | 48.9 | 0 | 27.5 | 0 | 3.8 | 0 | 0 | 31.7 | 0 | 100 | 0 | 96.9 | 0 | 38.6 | 0 | — | — | |
| PREFILLING-DLLMBase Model=Dream-v0-Base-7B, Sparsity=inter + intra-sparsity, B=5122026.06 | 18.02 | — | — | — | — | — | — | — | — | — | — | — | 24.57 | — | 95.42 | — | 70 | — | 43.62 | — | 0 | 53.71 | |
| PBSModel backbone=Qwen3-8B2026.02 | 17.2 | 18 | 77.3 | 1.9 | 74.68 | 47.6 | 74.73 | 30.5 | 76.49 | 3.8 | 74.24 | 78.06 | 33.7 | 70.98 | 100 | 71.75 | 100 | 72.09 | 39.2 | 75.04 | — | — | |
| FlexPrefillModel backbone=Qwen3-8B2026.02 | 17 | 14.9 | 82.5 | 1.9 | 82.66 | 45 | 84.94 | 16 | 83.29 | 3.9 | 83.28 | 85.81 | 34 | 81.64 | 99.8 | 81.81 | 99 | 82.37 | 36.8 | 83.14 | — | — | |
| Fast-dLLMBase Model=Dream-v0-Base-7B2026.06 | 8.38 | — | — | — | — | — | — | — | — | — | — | — | 3.43 | — | 0.17 | — | 3.05 | — | 3.96 | — | 0 | 8.73 | |
| YaRNBase Model=Dream-v0-Base-7B, Context Window=128K2026.06 | 7.87 | — | — | — | — | — | — | — | — | — | — | — | 3.43 | — | 0 | — | 0 | — | 2.9 | — | 0 | 6.11 | |
| Sparse-dLLMBase Model=Dream-v0-Base-7B, B=5122026.06 | 3.81 | — | — | — | — | — | — | — | — | — | — | — | 3.43 | — | 39.32 | — | 25.59 | — | 12.24 | — | 0 | 1.31 | |
| dKV-CacheBase Model=Dream-v0-Base-7B, B=5122026.06 | 3.05 | — | — | — | — | — | — | — | — | — | — | — | 4 | — | 0.51 | — | 0 | — | 2.21 | — | 0 | 5.68 | |
| Sparse-dLLMBase Model=UltraLLaDA, B=5122026.06 | 1.52 | — | — | — | — | — | — | — | — | — | — | — | 7.43 | — | 8.14 | — | 2.37 | — | 4.12 | — | 0 | 5.24 | |
| PREFILLING-DLLMBase Model=UltraLLaDA, Sparsity=inter + intra-sparsity, B=5122026.06 | 1.52 | — | — | — | — | — | — | — | — | — | — | — | 26 | — | 13.22 | — | 30 | — | 14.19 | — | 0 | 14.41 | |
| VanillaBase Model=UltraLLaDA, Context Window=128K2026.06 | 0.85 | — | — | — | — | — | — | — | — | — | — | — | 19.15 | — | 10.81 | — | 1.72 | — | 7.1 | — | 0 | 10.09 | |
| Fast-dLLMBase Model=UltraLLaDA2026.06 | 0.25 | — | — | — | — | — | — | — | — | — | — | — | 14.29 | — | 16.1 | — | 6.61 | — | 7.52 | — | 0 | 7.86 | |
| dKV-CacheBase Model=UltraLLaDA, B=5122026.06 | 0.25 | — | — | — | — | — | — | — | — | — | — | — | 16 | — | 5.59 | — | 7.12 | — | 6.21 | — | 0 | 8.3 | |
| VanillaBase Model=Dream-v0-Base-7B, Context Window=2K2026.06 | 0 | — | — | — | — | — | — | — | — | — | — | — | 34.57 | — | 1.69 | — | 2.54 | — | 12.44 | — | 0 | 35.81 |