# Learning 3D Semantic Segmentation with only 2D Image Supervision

Source: [Learning 3D Semantic Segmentation with only 2D Image Supervision](<https://www.sota2.com/research/paper/68fa6b0f1e209131cea2bd21>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:36:29.675Z

Authors: Kyle Genova, Xiaoqi Yin, Abhijit Kundu, Caroline Pantofaru, Forrester Cole, Avneesh Sud, Brian Brewington, Brian Shucker, Thomas Funkhouser

Published: 2021-10-21T17:56:28.000Z

## Abstract

With the recent growth of urban mapping and autonomous driving efforts, there has been an explosion of raw 3D data collected from terrestrial platforms with lidar scanners and color cameras. However, due to high labeling costs, ground-truth 3D semantic segmentation annotations are limited in both quantity and geographic diversity, while also being difficult to transfer across sensors. In contrast, large image collections with ground-truth semantic segmentations are readily available for diverse sets of scenes. In this paper, we investigate how to use only those labeled 2D image collections to supervise training 3D semantic segmentation models. Our approach is to train a 3D model from pseudo-labels derived from 2D semantic image segmentations using multiview fusion. We address several novel issues with this approach, including how to select trusted pseudo-labels, how to sample 3D scenes with rare object categories, and how to decouple input features from 2D images from pseudo-labels during training. The proposed network architecture, 2D3DNet, achieves significantly better performance (+6.2-11.4 mIoU) than baselines during experiments on a new urban dataset with lidar and images captured in 20 cities across 5 continents.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Semantic Segmentation on nuScenes 1.0 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-nuscenes-1-0-val.md>): Semantic segmentation; nuScenes 1.0 (val); mIoU: 79.

- [3D Semantic Segmentation on nuScenes Lidar-Seg (val)](<https://www.sota2.com/research/sota/3d-semantic-segmentation-on-nuscenes-lidar-seg-val.md>): 3D Semantic Segmentation; nuScenes Lidar-Seg (val); mIoU: 79.

- [3D Semantic Segmentation on ScanNet](<https://www.sota2.com/research/sota/3d-semantic-segmentation-on-scannet.md>): 3D Semantic Segmentation; ScanNet; Semantics mIoU: 55.23.

- [LiDAR Semantic Segmentation on nuScenes (val)](<https://www.sota2.com/research/sota/lidar-semantic-segmentation-on-nuscenes-val.md>): LiDAR Semantic Segmentation; nuScenes (val); mIoU: 79.

- [3D Semantic Segmentation on nuScenes (test)](<https://www.sota2.com/research/sota/3d-semantic-segmentation-on-nuscenes-test.md>): 3D Semantic Segmentation; nuScenes (test); mIoU: 80.

- [Semantic Segmentation on nuScenes (test)](<https://www.sota2.com/research/sota/semantic-segmentation-on-nuscenes-test.md>): Semantic segmentation; nuScenes (test); mIoU: 80.

- [LiDAR Semantic Segmentation on nuScenes official (test)](<https://www.sota2.com/research/sota/lidar-semantic-segmentation-on-nuscenes-official-test.md>): LiDAR Semantic Segmentation; nuScenes official (test); mIoU: 80.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2110.11325>)

[PDF](<https://arxiv.org/pdf/2110.11325.pdf>)