# DFormer: Diffusion-guided Transformer for Universal Image Segmentation

Source: [DFormer: Diffusion-guided Transformer for Universal Image Segmentation](<https://www.sota2.com/research/paper/68fa6f9e1e209131cea46a7d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:23:24.960Z

Authors: Hefeng Wang, Jiale Cao, Rao Muhammad Anwer, Jin Xie, Fahad Shahbaz Khan, Yanwei Pang

Published: 2023-06-06T06:33:32.000Z

## Abstract

This paper introduces an approach, named DFormer, for universal image segmentation. The proposed DFormer views universal image segmentation task as a denoising process using a diffusion model. DFormer first adds various levels of Gaussian noise to ground-truth masks, and then learns a model to predict denoising masks from corrupted masks. Specifically, we take deep pixel-level features along with the noisy masks as inputs to generate mask features and attention masks, employing diffusion-based decoder to perform mask prediction gradually. At inference, our DFormer directly predicts the masks and corresponding categories from a set of randomly-generated masks. Extensive experiments reveal the merits of our proposed contributions on different image segmentation tasks: panoptic segmentation, instance segmentation, and semantic segmentation. Our DFormer outperforms the recent diffusion-based panoptic segmentation method Pix2Seq-D with a gain of 3.6% on MS COCO val2017 set. Further, DFormer achieves promising semantic segmentation performance outperforming the recent diffusion-based method by 2.2% on ADE20K val set. Our source code and models will be publicly on https://github.com/cp3wan/DFormer

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Semantic Segmentation on SUN RGB-D (test)](<https://www.sota2.com/research/sota/semantic-segmentation-on-sun-rgb-d-test.md>): Semantic segmentation; SUN RGB-D (test); mIoU: 41.03.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2306.03437>)

[PDF](<https://arxiv.org/pdf/2306.03437.pdf>)