# MaterialPicker: Multi-Modal DiT-Based Material Generation

Source: [MaterialPicker: Multi-Modal DiT-Based Material Generation](<https://www.sota2.com/research/paper/68fa7a3b1e209131cea81f3f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:15:13.470Z

Authors: Xiaohe Ma, Valentin Deschaintre, Milo\\v{s} Ha\\v{s}an, Fujun Luan, Kun Zhou, Hongzhi Wu, Yiwei Hu

Published: 2024-12-04T11:23:15.000Z

## Abstract

High-quality material generation is key for virtual environment authoring and inverse rendering. We propose MaterialPicker, a multi-modal material generator leveraging a Diffusion Transformer (DiT) architecture, improving and simplifying the creation of high-quality materials from text prompts and/or photographs. Our method can generate a material based on an image crop of a material sample, even if the captured surface is distorted, viewed at an angle or partially occluded, as is often the case in photographs of natural scenes. We further allow the user to specify a text prompt to provide additional guidance for the generation. We finetune a pre-trained DiT-based video generator into a material generator, where each material map is treated as a frame in a video sequence. We evaluate our approach both quantitatively and qualitatively and show that it enables more diverse material generation and better distortion correction than previous work.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Material Extraction on Polyhaven Synthetic (test)](<https://www.sota2.com/research/sota/material-extraction-on-polyhaven-synthetic-test.md>): Material Extraction; Polyhaven Synthetic (test); LPIPS: 0.408.

- [Material Extraction on Polyhaven Real-World (test)](<https://www.sota2.com/research/sota/material-extraction-on-polyhaven-real-world-test.md>): Material Extraction; Polyhaven Real-World (test); LPIPS: 0.455.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2412.03225>)

[PDF](<https://arxiv.org/pdf/2412.03225.pdf>)
