# Open-vocabulary Object Segmentation with Diffusion Models

Source: [Open-vocabulary Object Segmentation with Diffusion Models](<https://www.sota2.com/research/paper/68fa70491e209131cea4a33e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:38:08.126Z

Authors: Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie

Published: 2023-01-12T18:59:08.000Z

## Abstract

The goal of this paper is to extract the visual-language correspondence from a pre-trained text-to-image diffusion model, in the form of segmentation map, i.e., simultaneously generating images and segmentation masks for the corresponding visual entities described in the text prompt. We make the following contributions: (i) we pair the existing Stable Diffusion model with a novel grounding module, that can be trained to align the visual and textual embedding space of the diffusion model with only a small number of object categories; (ii) we establish an automatic pipeline for constructing a dataset, that consists of {image, segmentation mask, text prompt} triplets, to train the proposed grounding module; (iii) we evaluate the performance of open-vocabulary grounding on images generated from the text-to-image diffusion model and show that the module can well segment the objects of categories beyond seen ones at training time; (iv) we adopt the augmented diffusion model to build a synthetic semantic segmentation dataset, and show that, training a standard segmentation model on such dataset demonstrates competitive performance on the zero-shot segmentation(ZS3) benchmark, which opens up new opportunities for adopting the powerful diffusion model for discriminative tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mask Generation on COCO-cap](<https://www.sota2.com/research/sota/mask-generation-on-coco-cap.md>): Mask Generation; COCO-cap; mIoU: 50.2.

- [Mask Generation on VOC sim (mIoU)](<https://www.sota2.com/research/sota/mask-generation-on-voc-sim-miou.md>): Mask Generation; VOC sim; mIoU: 62.1.

- [Semantic Segmentation on VOC 2012 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-voc-2012-val.md>): Semantic segmentation; VOC 2012 (val); mIoU: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2301.05221>)

[PDF](<https://arxiv.org/pdf/2301.05221.pdf>)
