# Tracking through Containers and Occluders in the Wild Source: [Tracking through Containers and Occluders in the Wild]() Collection index: [llms.txt]() Data updated: 2026-01-01T17:05:51.109Z Authors: Basile Van Hoorick, Pavel Tokmakov, Simon Stent, Jie Li, Carl Vondrick Published: 2023-05-04T17:59:58.000Z ## Abstract Tracking objects with persistence in cluttered and dynamic environments remains a difficult challenge for computer vision systems. In this paper, we introduce $\\textbf{TCOW}$, a new benchmark and model for visual tracking through heavy occlusion and containment. We set up a task where the goal is to, given a video sequence, segment both the projected extent of the target object, as well as the surrounding container or occluder whenever one exists. To study this task, we create a mixture of synthetic and annotated real datasets to support both supervised learning and structured evaluation of model performance under various forms of task variation, such as moving or nested containment. We evaluate two recent transformer-based video models and find that while they can be surprisingly capable of tracking targets under certain settings of task variation, there remains a considerable performance gap before we can claim a tracking model to have acquired a true notion of object permanence. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Video Object Segmentation on Rubric Office real-world](): Video Object Segmentation; Rubric Office real-world; Jtarget Score: 72.5. - [Video Object Segmentation on Rubric DAV/YTB (real-world)](): Video Object Segmentation; Rubric DAV/YTB (real-world); Jtarget: 52.8. - [Video Object Segmentation on TCOW Kubric Random synthetic (test)](): Video Object Segmentation; TCOW Kubric Random synthetic (test); Itgt Score (All): 53. - [Video Object Segmentation on TCOW Kubric Containers synthetic (test)](): Video Object Segmentation; TCOW Kubric Containers synthetic (test); Target IoU (All): 36.8. - [Video Object Segmentation on Rubric Cup Games (real-world)](): Video Object Segmentation; Rubric Cup Games (real-world); Jtarget: 38.3. - [Amodal Bounding Box Detection on TAO-Amodal custom 100-clip 1.0](): Amodal Bounding Box Detection; TAO-Amodal custom 100-clip 1.0; AP@25: 27.8. ## Original sources and implementation [Original paper]() [PDF]()