# Taming Outlier Tokens in Diffusion Transformers

Source: [Taming Outlier Tokens in Diffusion Transformers](<https://www.sota2.com/research/paper/69fc1cc65497b3b670c887b0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-08T05:00:14.135Z

Authors: Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu, Liang-Chieh Chen, Zhe Gan, Chen Wei

Published: 2026-05-06T17:59:42.000Z

## Abstract

We study outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work has shown that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models remains underexplored. We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop internal outlier tokens, especially in intermediate layers. Moreover, simply masking high-norm tokens does not improve performance, indicating that the problem is not only caused by a few extreme values, but is more closely related to corrupted local patch semantics. To address this issue, we introduce Dual-Stage Registers (DSR), a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser. Across ImageNet and large-scale text-to-image generation, these interventions consistently reduce outlier artifacts and improve generation quality. Our results highlight outlier-token control as an important ingredient in building stronger DiTs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-to-Image Generation on DPG-Bench (DPG Metric)](<https://www.sota2.com/research/sota/text-to-image-generation-on-dpg-bench-dpg-metric.md>): Text-to-Image Generation; DPG-Bench; DPG Score: 75.4.

- [Text-to-Image Generation on GenEval](<https://www.sota2.com/research/sota/text-to-image-generation-on-geneval.md>): Text-to-Image Generation; GenEval; GenEval Score: 46.6.

- [Class-conditional Image Generation on ImageNet 256x256](<https://www.sota2.com/research/sota/class-conditional-image-generation-on-imagenet-256x256.md>): Class-conditional Image Generation; ImageNet 256x256; Inception Score (IS): 223.85.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.05206>)

[PDF](<https://arxiv.org/pdf/2605.05206.pdf>)
