# 2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud

Source: [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](<https://www.sota2.com/research/paper/68fa65a91e209131cea0bb4c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:32:37.817Z

Authors: Mengdan Feng, Sixing Hu, Marcelo Ang, Gim Hee Lee

Published: 2019-04-22T06:49:46.000Z

## Abstract

Large-scale point cloud generated from 3D sensors is more accurate than its image-based counterpart. However, it is seldom used in visual pose estimation due to the difficulty in obtaining 2D-3D image to point cloud correspondences. In this paper, we propose the 2D3D-MatchNet - an end-to-end deep network architecture to jointly learn the descriptors for 2D and 3D keypoint from image and point cloud, respectively. As a result, we are able to directly match and establish 2D-3D correspondences from the query image and 3D point cloud reference map for visual pose estimation. We create our Oxford 2D-3D Patches dataset from the Oxford Robotcar dataset with the ground truth camera poses and 2D-3D image to point cloud correspondences for training and testing the deep network. Experimental results verify the feasibility of our approach.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image-to-point cloud registration on Oxford (test)](<https://www.sota2.com/research/sota/image-to-point-cloud-registration-on-oxford-test.md>): Image-to-point cloud registration; Oxford (test); RTE (m): 1.41.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1904.09742>)

[PDF](<https://arxiv.org/pdf/1904.09742.pdf>)