# Material Palette: Extraction of Materials from a Single Image

Source: [Material Palette: Extraction of Materials from a Single Image](<https://www.sota2.com/research/paper/68fa71461e209131cea5055d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:02.657Z

Authors: Ivan Lopes, Fabio Pizzati, Raoul de Charette

Published: 2023-11-28T18:59:58.000Z

## Abstract

In this paper, we propose a method to extract physically-based rendering (PBR) materials from a single real-world image. We do so in two steps: first, we map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be used in rendering applications. Our approach builds on existing synthetic material libraries with SVBRDF ground truth, but also exploits a diffusion-generated RGB texture dataset to allow generalization to new samples using unsupervised domain adaptation (UDA). Our contributions are thoroughly evaluated on synthetic and real-world datasets. We further demonstrate the applicability of our method for editing 3D scenes with materials estimated from real photographs. The code and models will be made open-source. Project page: https://astra-vision.github.io/MaterialPalette/

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Material Decomposition on Material Decomposition Dataset](<https://www.sota2.com/research/sota/material-decomposition-on-material-decomposition-dataset.md>): Material Decomposition; Material Decomposition Dataset; MSE (basecolor): 0.058.

- [Image-to-Material Generation on Single Real-world Images](<https://www.sota2.com/research/sota/image-to-material-generation-on-single-real-world-images.md>): Image-to-Material Generation; Single real-world images; CLIP Score (Basecolor): 0.813.

- [Material Extraction on Polyhaven Synthetic (test)](<https://www.sota2.com/research/sota/material-extraction-on-polyhaven-synthetic-test.md>): Material Extraction; Polyhaven Synthetic (test); LPIPS: 0.477.

- [Material Extraction on Polyhaven Real-World (test)](<https://www.sota2.com/research/sota/material-extraction-on-polyhaven-real-world-test.md>): Material Extraction; Polyhaven Real-World (test); LPIPS: 0.581.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2311.17060>)

[PDF](<https://arxiv.org/pdf/2311.17060.pdf>)
