# TAPVid-3D: A Benchmark for Tracking Any Point in 3D

Source: [TAPVid-3D: A Benchmark for Tracking Any Point in 3D](<https://www.sota2.com/research/paper/68fa74a51e209131cea62e79>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:14:35.342Z

Authors: Skanda Koppula, Ignacio Rocco, Yi Yang, Joe Heyward, Jo\\~ao Carreira, Andrew Zisserman, Gabriel Brostow, Carl Doersch

Published: 2024-07-08T13:28:47.000Z

## Abstract

We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [3D Point Tracking on TAPVid-3D (minival)](<https://www.sota2.com/research/sota/3d-point-tracking-on-tapvid-3d-minival.md>): 3D Point Tracking; TAPVid-3D (minival); Aria 3D-AJ: 8.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2407.05921>)

[PDF](<https://arxiv.org/pdf/2407.05921.pdf>)