SpatialTracker: Tracking Any 2D Pixels in 3D Space
About
Recovering dense and long-range pixel motion in videos is a challenging problem. Part of the difficulty arises from the 3D-to-2D projection process, leading to occlusions and discontinuities in the 2D motion domain. While 2D motion can be intricate, we posit that the underlying 3D motion can often be simple and low-dimensional. In this work, we propose to estimate point trajectories in 3D space to mitigate the issues caused by image projection. Our method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth estimators, represents the 3D content of each frame efficiently using a triplane representation, and performs iterative updates using a transformer to estimate 3D trajectories. Tracking in 3D allows us to leverage as-rigid-as-possible (ARAP) constraints while simultaneously learning a rigidity embedding that clusters pixels into different rigid parts. Extensive evaluation shows that our approach achieves state-of-the-art tracking performance both qualitatively and quantitatively, particularly in challenging scenarios such as out-of-plane rotation.
Key insights
What problem does SpatialTracker solve?
It tracks dense, long-range pixel motion by lifting 2D pixels into 3D using monocular depth, then estimating 3D trajectories with a transformer and triplane features to handle occlusion and complex motion better.
How does it differ from prior tracking methods?
Unlike prior 2D-correlation trackers, it performs 3D feature correlation with as-rigid-as-possible regularization. On TAP-Vid-DAVIS it achieves 76.3 average position accuracy versus CoTracker's 75.4.
What are the practical uses of these findings?
The method produces accurate long-range 2D and 3D trajectories for challenging real-world videos, plus self-supervised rigid part segmentation.
What limitations do the authors note?
Tracking performance depends on off-the-shelf monocular depth estimator accuracy; Table 5 shows performance varies across depth models. The authors expect future depth estimation advances to improve results.