# OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning

Source: [OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning](<https://www.sota2.com/research/paper/68fa77b71e209131cea74ef2>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:42:31.645Z

Authors: Anwesa Choudhuri, Girish Chowdhary, Alexander G. Schwing

Published: 2024-04-04T17:59:58.000Z

## Abstract

We propose the new task 'open-world video instance segmentation and captioning'. It requires to detect, segment, track and describe with rich captions never before seen objects. This challenging task can be addressed by developing "abstractors" which connect a vision model and a language foundation model. Concretely, we connect a multi-scale visual feature extractor and a large language model (LLM) by developing an object abstractor and an object-to-text abstractor. The object abstractor, consisting of a prompt encoder and transformer blocks, introduces spatially-diverse open-world object queries to discover never before seen objects in videos. An inter-query contrastive loss further encourages the diversity of object queries. The object-to-text abstractor is augmented with masked cross-attention and acts as a bridge between the object queries and a frozen LLM to generate rich and descriptive object-centric captions for each detected object. Our generalized approach surpasses the baseline that jointly addresses the tasks of open-world video instance segmentation and dense video object captioning by 13% on never before seen objects, and by 10% on object-centric captions.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Dense Video Object Captioning on VidSTG (test)](<https://www.sota2.com/research/sota/dense-video-object-captioning-on-vidstg-test.md>): Dense Video Object Captioning; VidSTG (test); CapA: 43.9.

- [Open-world tracking on BURST (test)](<https://www.sota2.com/research/sota/open-world-tracking-on-burst-test.md>): Open-world tracking; BURST (test); Unseen Performance: 57.2.

- [Dense Video Object Captioning on VidSTG](<https://www.sota2.com/research/sota/dense-video-object-captioning-on-vidstg.md>): Dense Video Object Captioning; VidSTG; CapA Score: 43.9.

- [Open-World Video Segmentation on BURST (val)](<https://www.sota2.com/research/sota/open-world-video-segmentation-on-burst-val.md>): Open-World Video Segmentation; BURST (val); OWTA (All): 69.

- [Video Multimodal Interpretation on OVIS (val)](<https://www.sota2.com/research/sota/video-multimodal-interpretation-on-ovis-val.md>): Video Multimodal Interpretation; OVIS (val); AP: 25.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.03657>)

[PDF](<https://arxiv.org/pdf/2404.03657.pdf>)

[Implementation](<https://anwesachoudhuri.github.io/OpenWorldVISCap/>)