# OneFormer: One Transformer to Rule Universal Image Segmentation

Source: [OneFormer: One Transformer to Rule Universal Image Segmentation](<https://www.sota2.com/research/paper/68fa6e4a1e209131cea3f0c1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:12:56.721Z

Authors: Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi

Published: 2022-11-10T18:56:04.000Z

## Abstract

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible. To support further research, we open-source our code and models at https://github.com/SHI-Labs/OneFormer

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Instance Segmentation on Cityscapes (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-cityscapes-val.md>): Instance Segmentation; Cityscapes (val); AP: 50.6.

- [Panoptic Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-ade20k-val.md>): Panoptic Segmentation; ADE20K (val); PQ: 54.5.

- [Semantic Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-coco-2017-val.md>): Semantic segmentation; COCO 2017 (val); mIoU: 68.1.

- [Panoptic Segmentation on ADE20K 15 (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-ade20k-15-val.md>): Panoptic Segmentation; ADE20K 15 (val); PQ: 51.5.

- [Instance Segmentation on ADE20K 15 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-ade20k-15-val.md>): Instance Segmentation; ADE20K 15 (val); AP: 37.8.

- [Panoptic Segmentation on Cityscapes (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-cityscapes-val.md>): Panoptic Segmentation; Cityscapes (val); PQ: 70.6.

- [Semantic Segmentation on COCO (mIoU, Acc)](<https://www.sota2.com/research/sota/semantic-segmentation-on-coco-miou-acc.md>): Semantic segmentation; COCO; mIoU: 65.52.

- [Semantic Segmentation on Cityscapes 18 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-18-val.md>): Semantic segmentation; Cityscapes 18 (val); mIoU (small-scale): 83.

- [Panoptic Segmentation on COCO Panoptic 2017 (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-coco-panoptic-2017-val.md>): Panoptic Segmentation; COCO Panoptic 2017 (val); PQ: 57.9.

- [Instance Segmentation on ADE20K 150 classes (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-ade20k-150-classes-val.md>): Instance Segmentation; ADE20K 150 59 (val); AP: 37.8.

- [Semantic Segmentation on Cityscapes (val) (mIoU s.s. and mIoU m.s.)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-val-miou-s-s-and-miou-m-s.md>): Semantic segmentation; Cityscapes (val); mIoU (small): 83.6.

- [Segmentation on COCO (in-domain)](<https://www.sota2.com/research/sota/segmentation-on-coco-in-domain.md>): Segmentation; COCO in-domain; PQ: 57.9.

- [Panoptic Segmentation on MUSES (test)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-muses-test.md>): Panoptic Segmentation; MUSES (test); PQ (Clear): 58.3.

- [Panoptic Segmentation on COCO (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-coco-val.md>): Panoptic Segmentation; COCO (val); PQ: 57.9.

- [Instance Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-ade20k-val.md>): Instance Segmentation; ADE20K (val); AP: 40.2.

- [Semantic Segmentation on Cityscapes (val) (Specific mIoU Variants)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-val-specific-miou-variants.md>): Semantic segmentation; Cityscapes (val); mIoU (single-scale): 85.1.

- [Semantic Segmentation on MUSES (test)](<https://www.sota2.com/research/sota/semantic-segmentation-on-muses-test.md>): Semantic segmentation; MUSES (test); mIoU: 72.8.

- [Semantic Segmentation on Cityscapes 17 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-17-val.md>): Semantic segmentation; Cityscapes 17 (val); mIoU: 83.6.

- [Semantic Segmentation on ADE20K 15 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-15-val.md>): Semantic segmentation; ADE20K 15 (val); mIoU (ss): 58.3.

- [Semantic Segmentation on ADE20K 2016 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-2016-val.md>): Semantic segmentation; ADE20K 2016 (val); mIoU: 58.1.

- [Semantic Segmentation on Cityscapes 13 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-13-val.md>): Semantic segmentation; Cityscapes 13 (val); mIoU: 83.6.

- [Semantic Segmentation on COCO (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-coco-val.md>): Semantic segmentation; COCO (val); mIoU: 67.4.

- [Panoptic Segmentation on MoNuSAC 2020 (test)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-monusac-2020-test.md>): Panoptic Segmentation; MoNuSAC 2020 (test); PQ: 31.35.

- [Semantic Segmentation on ADE20K 70 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-70-val.md>): Semantic segmentation; ADE20K 70 (val); mIoU: 58.1.

- [Semantic Segmentation on Cityscapes (val) (s.s. mIoU and m.s. mIoU)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-val-s-s-miou-and-m-s-miou.md>): Semantic segmentation; Cityscapes (val); mIoU (s.s.): 83.6.

- [Panoptic Segmentation on NuInsSeg](<https://www.sota2.com/research/sota/panoptic-segmentation-on-nuinsseg.md>): Panoptic Segmentation; NuInsSeg; PQ: 16.91.

- [Semantic Segmentation on ADE20K (mIoU)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-miou.md>): Semantic segmentation; ADE20K; mIoU: 57.

- [Panoptic Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-coco-2017-val.md>): Panoptic Segmentation; COCO 2017 (val); PQ: 58.

- [Semantic Segmentation on Cityscapes](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes.md>): Semantic segmentation; Cityscapes; mIoU: 83.6.

- [Semantic Segmentation on Cityscapes (val) (mIoU)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-val-miou.md>): Semantic segmentation; Cityscapes (val); mIoU: 85.8.

- [Semantic Segmentation on ADE20K](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k.md>): Semantic segmentation; ADE20K; mIoU: 57.4.

- [Instance Segmentation on COCO (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-val.md>): Instance Segmentation; COCO (val); APmk: 48.9.

- [Semantic Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-val.md>): Semantic segmentation; ADE20K (val); mIoU: 57.4.

- [Semantic Segmentation on Cityscapes (val) (mIoU only)](<https://www.sota2.com/research/sota/semantic-segmentation-on-cityscapes-val-miou-only.md>): Semantic segmentation; Cityscapes (val); mIoU: 80.3.

- [Instance Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017-val.md>): Instance Segmentation; COCO 2017 (val); APm: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2211.06220>)

[PDF](<https://arxiv.org/pdf/2211.06220.pdf>)

[Implementation](<https://github.com/SHI-Labs/OneFormer>)
