# Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention Source: [Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention]() Collection index: [llms.txt]() Data updated: 2026-05-21T05:00:13.694Z Authors: Wenhu Zhang, Yiming Wu, Huanyu Wang, Yaoyang Liu, Huanzhang Dou, Senqiao Yang, Sitong Wu, Hanbin Zhao, Jiaya Jia Published: 2026-05-19T12:01:13.000Z ## Abstract Diffusion Language Models (DLMs) enable globally coherent, bidirectional, and controllable text generation, offering advantages over traditional autoregressive LLMs, while scaling to ultra-long sequences remains costly. Many existing block-sparse attention methods select blocks by fixed sampling patterns over the high-resolution attention space, such as tail regions or anti-diagonal stripes. Such prior-driven sampling can miss salient tokens and introduce instability under distribution shifts. In this paper, we propose the Block Approximate Sparse Attention framework (BA-Att) with block-wise pre-downsampled operation, which identifies informative regions within a compact downsampled space, avoiding reliance on brittle positional priors. To analyze its theoretical behavior, we define an oracle post-downsample attention map and formalize the approximation error between pre- and post-downsample schemes. Based on this insight, we introduce a lightweight norm-sorting module and a covariance-compensated correction that approximates full covariance using diagonal QK variances, reducing computational complexity. Extensive experiments show that our operator achieves up to 6.95x acceleration over FlashAttention in attention computation, and maintains near full-attention performance at 50% sparsity across language models, multimodal language models, and video generation models, demonstrating strong efficiency and generalization. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Long-context Language Modeling on LongBench 16K context length](): Long-context language modeling; LongBench 16K context length; NrtvQA Score: 13.3. - [Long-context reasoning on RULER 8K context length](): Long-context Reasoning; RULER 8K context length; NIAH Score: 98.5. - [Long-context reasoning on RULER 16K context length](): Long-context Reasoning; RULER 16k context length; NIAH: 94.62. - [Video Generation on VBench (PSNR, SSIM, LPIPS)](): Video Generation; VBench; PSNR: 24.08. - [Long-context reasoning on RULER 4K context length](): Long-context Reasoning; RULER 4k context length; NIAH: 98.16. - [Video Understanding on MuirBench](): Video Understanding; MuirBench; Score: 47.69. - [Long-context reasoning on RULER 32K context length](): Long-context Reasoning; RULER 32k context length; NIAH: 90.97. - [Video Understanding on MLVU (dev)](): Video Understanding; MLVU (dev); MLVU Dev Score: 59.71. - [Video Understanding on VideoMME (Score %)](): Video Understanding; VideoMME; Score (%): 56.56. ## Original sources and implementation [Original paper]() [PDF]()