# FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models Source: [FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models]() Collection index: [llms.txt]() Data updated: 2026-06-11T05:00:05.512Z Authors: Sankeerth Durvasula, Kavya Sreedhar, Zain Moustafa, Suraj Kothawade, Tianlei Pang, Ashish Gondimalla, Suvinay Subramanian, Narges Shahidi, Nandita Vijaykumar Published: 2025-09-20T03:48:32.000Z ## Abstract Using diffusion transformers for media generation may require evaluating attention over extremely long sequences, with attention layers accounting for the majority of generation latency. Exploiting sparsity in attention maps offers a promising opportunity to reduce this cost. In this work, we show that attention maps in diffusion transformers exhibit significant fine-grained sparsity in video generation models. Existing sparse attention methods, however, are too coarse-grained, leaving a large fraction of redundant computation unaddressed, or incur high overheads at finer granularity. We propose FG-Attn, a novel, low-overhead fine-grained sparse attention mechanism that skips score computations at the granularity of a MxN tile, where N\>=1 and M\>=16, and where each block is the result of query-key dot products between M queries and N keys. FG-Attn addresses the key challenge of hardware underutilization in sparse attention kernels on GPUs, without incurring the overheads of irregular memory access and redundant operations. FG-Attn can fully supersede existing sparse attention methods and extend block sparse attention methods to finer granularities on modern GPUs. At 70% sparsity, FG-Attn is up to 2.45X faster than the state-of-art FlashInfer, and reduces attention kernel time by 14.7% on average. FG-Attn speeds up end-to-end video generation times by up to 1.40X (1.18X on average) over Flash Attention 3. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Video Generation on LTX2 1024x1536 121 Frames](): Video Generation; LTX2 1024x1536 121 Frames; Generation Time (s): 2. - [Video Generation on HunyuanVideo 720p 129 Frames](): Video Generation; HunyuanVideo 720p 129 Frames; SSIM: 0.81. - [Video Generation on LTX2 512x768, 121 Frames](): Video Generation; LTX2 512x768, 121 Frames; Generation Time (s): 0. - [Video Generation on HunyuanVideo 480p (129 Frames)](): Video Generation; HunyuanVideo 480p, 129 Frames; SSIM: 83. - [Video Generation on Wan 14B 480p 81 Frames 2.1](): Video Generation; Wan 14B 480p 81 Frames 2.1; SSIM: 87. - [Video Generation on Wan 14B 720p 81 Frames 2.1](): Video Generation; Wan 14B 720p 81 Frames 2.1; SSIM: 84. ## Original sources and implementation [Original paper]() [PDF]()