# Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation Source: [Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:01:03.352Z Authors: Hao Fang, Peng Wu, Yawei Li, Xinxin Zhang, Xiankai Lu Published: 2024-07-10T07:30:51.000Z ## Abstract Open-Vocabulary Video Instance Segmentation (VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, the recent Open-Vocabulary VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between the VLM features (e.g., CLIP) and the instance queries and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel Open-Vocabulary VIS baseline called OVFormer. OVFormer utilizes a lightweight module for unified embedding alignment between query embeddings and CLIP image embeddings to remedy the domain gap. Unlike previous image-based training methods, we conduct video-based model training and deploy a semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer achieves 21.9 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art performance by 7.7. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer (+ 7.6 mAP on YouTube-VIS 2019, + 3.9 mAP on OVIS). Code is available at https://github.com/fanghaook/OVFormer. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Video Multimodal Interpretation on LV-VIS (val)](): Video Multimodal Interpretation; LV-VIS (val); AP: 24.7. - [Controllable Video Segmentation and Captioning on Video Dataset](): Controllable Video Segmentation and Captioning; Video Dataset; FPS: 2.98. - [Dense Video Object Captioning on VidSTG](): Dense Video Object Captioning; VidSTG; CapA Score: 12.8. - [Video Multimodal Interpretation on OVIS (val)](): Video Multimodal Interpretation; OVIS (val); AP: 21.3. ## Original sources and implementation [Original paper]() [PDF]()