# Open Vocabulary Panoptic Segmentation With Retrieval Augmentation

Source: [Open Vocabulary Panoptic Segmentation With Retrieval Augmentation](<https://www.sota2.com/research/paper/69705d93e25ad2fa9d1ed152>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-22T05:00:21.476Z

Authors: Nafis Sadeq, Qingfeng Liu, Mostafa El-Khamy

Published: 2026-01-19T07:16:45.000Z

## Abstract

Given an input image and set of class names, panoptic segmentation aims to label each pixel in an image with class labels and instance labels. In comparison, Open Vocabulary Panoptic Segmentation aims to facilitate the segmentation of arbitrary classes according to user input. The challenge is that a panoptic segmentation system trained on a particular dataset typically does not generalize well to unseen classes beyond the training data. In this work, we propose RetCLIP, a retrieval-augmented panoptic segmentation method that improves the performance of unseen classes. In particular, we construct a masked segment feature database using paired image-text data. At inference time, we use masked segment features from the input image as query keys to retrieve similar features and associated class labels from the database. Classification scores for the masked segment are assigned based on the similarity between query features and retrieved features. The retrieval-based classification scores are combined with CLIP-based scores to produce the final output. We incorporate our solution with a previous SOTA method (FC-CLIP). When trained on COCO, the proposed method demonstrates 30.9 PQ, 19.3 mAP, 44.0 mIoU on the ADE20k dataset, achieving +4.5 PQ, +2.5 mAP, +10.0 mIoU absolute improvement over the baseline.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Panoptic Segmentation on ADE20K 150 (val)](<https://www.sota2.com/research/sota/panoptic-segmentation-on-ade20k-150-val.md>): Panoptic Segmentation; ADE20K 150 (val); PQ: 30.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.12779>)

[PDF](<https://arxiv.org/pdf/2601.12779.pdf>)