SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
About
We present SDXL, a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder. We design multiple novel conditioning schemes and train SDXL on multiple aspect ratios. We also introduce a refinement model which is used to improve the visual fidelity of samples generated by SDXL using a post-hoc image-to-image technique. We demonstrate that SDXL shows drastically improved performance compared the previous versions of Stable Diffusion and achieves results competitive with those of black-box state-of-the-art image generators. In the spirit of promoting open research and fostering transparency in large model training and evaluation, we provide access to code and model weights at https://github.com/Stability-AI/generative-models
Key insights
What problem does SDXL address and how?
It improves open text-to-image latent diffusion by using a 2.6B UNet, dual text encoders, size/crop conditioning, multi-aspect training, a better autoencoder, and an optional refinement stage.
How does SDXL differ from prior work?
SDXL adds size and crop conditioning and an optional refiner. In a four-model user study, participants chose refined SDXL 48.44% of the time, versus 7.91% for SD 1.5 and 6.71% for SD 2.1.
What can creators and researchers use SDXL for?
The released model generates high-resolution images from text, with support for varied aspect ratios. Its open code and weights give researchers a foundation to inspect, adapt and compare image-generation systems.
What limitations and future work do the authors note?
The two-stage pipeline increases memory and sampling cost. The model struggles with hands, complex prompts, concept bleeding, and long text. Future work targets single-stage models, distillation, and better text rendering.