# DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification

Source: [DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification](<https://www.sota2.com/research/paper/68fa73271e209131cea5ab8c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:43:27.234Z

Authors: Sitian Shen, Zilin Zhu, Linqian Fan, Harry Zhang, Xinxiao Wu

Published: 2023-05-25T11:55:38.000Z

## Abstract

Large pre-trained models have had a significant impact on computer vision by enabling multi-modal learning, where the CLIP model has achieved impressive results in image classification, object detection, and semantic segmentation. However, the model's performance on 3D point cloud processing tasks is limited due to the domain gap between depth maps from 3D projection and training images of CLIP. This paper proposes DiffCLIP, a new pre-training framework that incorporates stable diffusion with ControlNet to minimize the domain gap in the visual branch. Additionally, a style-prompt generation module is introduced for few-shot tasks in the textual branch. Extensive experiments on the ModelNet10, ModelNet40, and ScanObjectNN datasets show that DiffCLIP has strong abilities for 3D understanding. By using stable diffusion and style-prompt generation, DiffCLIP achieves an accuracy of 43.2\\% for zero-shot classification on OBJ\\\_BG of ScanObjectNN, which is state-of-the-art performance, and an accuracy of 80.6\\% for zero-shot classification on ModelNet10, which is comparable to state-of-the-art performance.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.15957>)

[PDF](<https://arxiv.org/pdf/2305.15957.pdf>)
