# Visual Geometry Grounded Deep Structure From Motion

Source: [Visual Geometry Grounded Deep Structure From Motion](<https://www.sota2.com/research/paper/68fa71661e209131cea512c9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:39:59.054Z

Authors: Jianyuan Wang, Nikita Karaev, Christian Rupprecht, David Novotny

Published: 2023-12-07T18:59:52.000Z

## Abstract

Structure-from-motion (SfM) is a long-standing problem in the computer vision community, which aims to reconstruct the camera poses and 3D structure of a scene from a set of unconstrained 2D images. Classical frameworks solve this problem in an incremental manner by detecting and matching keypoints, registering images, triangulating 3D points, and conducting bundle adjustment. Recent research efforts have predominantly revolved around harnessing the power of deep learning techniques to enhance specific elements (e.g., keypoint matching), but are still based on the original, non-differentiable pipeline. Instead, we propose a new deep pipeline VGGSfM, where each component is fully differentiable and thus can be trained in an end-to-end manner. To this end, we introduce new mechanisms and simplifications. First, we build on recent advances in deep 2D point tracking to extract reliable pixel-accurate tracks, which eliminates the need for chaining pairwise matches. Furthermore, we recover all cameras simultaneously based on the image and track features instead of gradually registering cameras. Finally, we optimise the cameras and triangulate 3D points via a differentiable bundle adjustment layer. We attain state-of-the-art performance on three popular datasets, CO3D, IMC Phototourism, and ETH3D.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [3D Triangulation on ETH3D](<https://www.sota2.com/research/sota/3d-triangulation-on-eth3d.md>): 3D Triangulation; ETH3D; Accuracy (1cm): 80.62.

- [Multi-view camera pose estimation on IMC PhotoTourism](<https://www.sota2.com/research/sota/multi-view-camera-pose-estimation-on-imc-phototourism.md>): Multi-View Camera Pose Estimation; IMC PhotoTourism; AUC @ 3° Threshold: 45.23.

- [3D Triangulation on ETH3D (train)](<https://www.sota2.com/research/sota/3d-triangulation-on-eth3d-train.md>): 3D Triangulation; ETH3D (train); Accuracy (1cm): 80.62.

- [Camera Pose Estimation on IMC](<https://www.sota2.com/research/sota/camera-pose-estimation-on-imc.md>): Camera pose estimation; IMC; AUC (3° Threshold): 0.4523.

- [Multi-View Camera Pose Estimation on IMC Dataset](<https://www.sota2.com/research/sota/multi-view-camera-pose-estimation-on-imc-dataset.md>): Multi-View Camera Pose Estimation; IMC Dataset; AUC @ 3°: 45.23.

- [Camera Pose Estimation on Co3D (test)](<https://www.sota2.com/research/sota/camera-pose-estimation-on-co3d-test.md>): Camera pose estimation; Co3D (test); RRE @ 15°: 88.2.

- [Multi-view camera pose estimation on Texture-Poor SfM](<https://www.sota2.com/research/sota/multi-view-camera-pose-estimation-on-texture-poor-sfm.md>): Multi-View Camera Pose Estimation; Texture-Poor SfM; AUC @ 3°: 1.86.

- [Multi-View Camera Pose Estimation on ETH3D](<https://www.sota2.com/research/sota/multi-view-camera-pose-estimation-on-eth3d.md>): Multi-View Camera Pose Estimation; ETH3D; AUC@1°: 0.2809.

- [Multi-View Camera Pose Estimation on Texture-Poor SfM Dataset](<https://www.sota2.com/research/sota/multi-view-camera-pose-estimation-on-texture-poor-sfm-dataset.md>): Multi-View Camera Pose Estimation; Texture-Poor SfM Dataset; AUC (Threshold 3°): 1.86.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.04563>)

[PDF](<https://arxiv.org/pdf/2312.04563.pdf>)
