# LightIt: Illumination Modeling and Control for Diffusion Models

Source: [LightIt: Illumination Modeling and Control for Diffusion Models](<https://www.sota2.com/research/paper/68fa729a1e209131cea576ee>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T05:16:22.046Z

Authors: Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nie{\\ss}ner, Yannick Hold-Geoffroy (2) \_\_INSTITUTION\_5\_\_ Technical University of Munich, (2) Adobe Research)

Published: 2024-03-15T18:26:33.000Z

## Abstract

We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Relighting on Outdoor Laval (test)](<https://www.sota2.com/research/sota/image-relighting-on-outdoor-laval-test.md>): Image Relighting; Outdoor Laval (test); FID: 64.18.

- [Relighting Evaluation on Internal Cycle Relighting (test)](<https://www.sota2.com/research/sota/relighting-evaluation-on-internal-cycle-relighting-test.md>): Relighting Evaluation; Internal Cycle Relighting (test); PSNR: 20.44.

- [Perceptual Image Generation Quality on Internal User Study (test)](<https://www.sota2.com/research/sota/perceptual-image-generation-quality-on-internal-user-study-test.md>): Perceptual Image Generation Quality; Internal User Study (test); L-PQ: 95.57.

- [Indoor Scene Relighting on LSUN Bedrooms (test)](<https://www.sota2.com/research/sota/indoor-scene-relighting-on-lsun-bedrooms-test.md>): Indoor Scene Relighting; LSUN Bedrooms (test); RMSE: 0.227.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.10615>)

[PDF](<https://arxiv.org/pdf/2403.10615.pdf>)
