# UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models Source: [UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models]() Collection index: [llms.txt]() Data updated: 2025-10-27T07:04:32.834Z Authors: Guangxin He, Shen Nie, Fengqi Zhu, Yuankang Zhao, Tianyi Bai, Ran Yan, Jie Fu, Chongxuan Li, Binhang Yuan Published: 2025-10-12T07:26:56.000Z ## Abstract Diffusion LLMs have attracted growing interest, with plenty of recent work emphasizing their great potential in various downstream tasks; yet the long-context behavior of diffusion LLMs remains largely uncharted. We present a case study of post-training techniques for extending the context window of diffusion LLMs (i.e., LLaDA) without retraining from scratch. We show that a simple modification to the standard Rotary Positional Embeddings (RoPE) extension effectively accommodates the probabilistic modeling inherent in the diffusion process, enabling stable scaling to longer context ranges. We further compare masking strategies used during post-training and analyze their impact on optimization stability and long-range recall. Instantiating these insights, we introduce UltraLLaDA, a diffusion LLM with a 128K-token context window that, in our empirical evaluation on long-context tasks, significantly outperforms training-free baselines. Our experimental results highlight the special positional extension as a key lever for scaling diffusion LLMs to extended contexts and offer practical guidance for practitioners seeking 128K-scale context via efficient post-training. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Long-context reasoning on RULER 32K context length](): Long-context Reasoning; RULER 32k context length; NIAH: 92.78. - [Long-context Language Modeling on LongBench 16K context length](): Long-context language modeling; LongBench 16K context length; NrtvQA Score: 12.1. - [Long-context reasoning on RULER 8K context length](): Long-context Reasoning; RULER 8K context length; NIAH Score: 98.28. - [Long-context reasoning on RULER 16K context length](): Long-context Reasoning; RULER 16k context length; NIAH: 93. - [Long-context reasoning on RULER 4K context length](): Long-context Reasoning; RULER 4k context length; NIAH: 97.31. ## Original sources and implementation [Original paper]() [PDF]()