Randomized 3D Scene Generation for Generalizable Self-Supervised Pre-Training

Capturing and labeling real-world 3D data is laborious and time-consuming, which makes it costly to train strong 3D models. To address this issue, recent works present a simple method by generating randomized 3D scenes without simulation and rendering. Although models pre-trained on the generated synthetic data gain impressive performance boosts, previous works have two major shortcomings. First, they focus on only one downstream task (i.e., object detection), and the generalization to other tasks is unexplored. Second, the contributions of generated data are not systematically studied. To obtain a deeper understanding of the randomized 3D scene generation technique, we revisit previous works and compare different data generation methods using a unified setup. Moreover, to clarify the generalization of the pre-trained models, we evaluate their performance in multiple tasks (i.e., object detection and semantic segmentation) and with different pre-training methods (i.e., masked autoencoder and contrastive learning). Moreover, we propose a new method to generate 3D scenes with spherical harmonics. It surpasses the previous formula-driven method with a clear margin and achieves on-par results with methods using real-world scans and CAD models.

Lanxiao Li, Michael Heizmann · 2023