# LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation

Source: [LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation](<https://www.sota2.com/research/paper/68fa7bac1e209131cea89ce1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:46:25.721Z

Authors: Jiahao Wang, Ning Kang, Lewei Yao, Mengzhao Chen, Chengyue Wu, Songyang Zhang, Shuchen Xue, Yong Liu, Taiqiang Wu, Xihui Liu, Kaipeng Zhang, Shifeng Zhang, Wenqi Shao, Zhenguo Li, Ping Luo

Published: 2025-01-22T16:02:06.000Z

## Abstract

In this paper, we investigate how to convert a pre-trained Diffusion Transformer (DiT) into a linear DiT, as its simplicity, parallelism, and efficiency for image generation. Through detailed exploration, we offer a suite of ready-to-use solutions, ranging from linear attention design to optimization strategies. Our core contributions include 5 practical guidelines: 1) Applying depth-wise convolution within simple linear attention is sufficient for image generation. 2) Using fewer heads in linear attention provides a free-lunch performance boost without increasing latency. 3) Inheriting weights from a fully converged, pre-trained DiT. 4) Loading all parameters except those related to linear attention. 5) Hybrid knowledge distillation: using a pre-trained teacher DiT to help the training of the student linear DiT, supervising not only the predicted noise but also the variance of the reverse diffusion process. These guidelines lead to our proposed \\underline{L}inear D\\underline{i}ffusion \\underline{T}ransformer (LiT), which serves as a safe and efficient alternative baseline for DiT with pure linear attention. In class-conditional 256$\\times$256 and 512$\\times$512 ImageNet generation, LiT can be quickly adapted from DiT using only $20\\%$ and $33\\%$ of DiT's training steps, respectively, while achieving comparable performance. LiT also rivals methods based on Mamba or Gated Linear Attention. Moreover, the same guidelines generalize to text-to-image generation: LiT can be swiftly converted from PixArt-$\\Sigma$ to generate high-quality images, maintaining comparable GenEval scores.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Generation on ImageNet-1K 256x256 (val)](<https://www.sota2.com/research/sota/image-generation-on-imagenet-1k-256x256-val.md>): Image Generation; ImageNet-1K 256x256 (val); Inception Score: 22.08.

- [Class-conditional Image Generation on ImageNet 256x256](<https://www.sota2.com/research/sota/class-conditional-image-generation-on-imagenet-256x256.md>): Class-conditional Image Generation; ImageNet 256x256; Inception Score (IS): 265.2.

- [Image Generation on ImageNet (val)](<https://www.sota2.com/research/sota/image-generation-on-imagenet-val.md>): Image Generation; ImageNet (val); Inception Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2501.12976>)

[PDF](<https://arxiv.org/pdf/2501.12976.pdf>)
