# Collaborative Multi-Modal Coding for High-Quality 3D Generation

Source: [Collaborative Multi-Modal Coding for High-Quality 3D Generation](<https://www.sota2.com/research/paper/68fa7ad41e209131cea855d9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-20T05:00:13.953Z

Authors: Ziang Cao, Zhaoxi Chen, Liang Pan, Ziwei Liu

Published: 2025-08-21T04:31:14.000Z

## Abstract

3D content inherently encompasses multi-modal characteristics and can be projected into different modalities (e.g., RGB images, RGBD, and point clouds). Each modality exhibits distinct advantages in 3D asset modeling: RGB images contain vivid 3D textures, whereas point clouds define fine-grained 3D geometries. However, most existing 3D-native generative architectures either operate predominantly within single-modality paradigms-thus overlooking the complementary benefits of multi-modality data-or restrict themselves to 3D structures, thereby limiting the scope of available training datasets. To holistically harness multi-modalities for 3D modeling, we present TriMM, the first feed-forward 3D-native generative model that learns from basic multi-modalities (e.g., RGB, RGBD, and point cloud). Specifically, 1) TriMM first introduces collaborative multi-modal coding, which integrates modality-specific features while preserving their unique representational strengths. 2) Furthermore, auxiliary 2D and 3D supervision are introduced to raise the robustness and performance of multi-modal coding. 3) Based on the embedded multi-modal code, TriMM employs a triplane latent diffusion model to generate 3D assets of superior quality, enhancing both the texture and the geometric detail. Extensive experiments on multiple well-known datasets demonstrate that TriMM, by effectively leveraging multi-modality, achieves competitive performance with models trained on large-scale datasets, despite utilizing a small amount of training data. Furthermore, we conduct additional experiments on recent RGB-D datasets, verifying the feasibility of incorporating other multi-modal datasets into 3D generation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [3D Reconstruction on OmniObject3D (1700 unseen samples)](<https://www.sota2.com/research/sota/3d-reconstruction-on-omniobject3d-1700-unseen-samples.md>): 3D Reconstruction; OmniObject3D (1700 unseen samples); CLIP Score: 0.674.

- [3D Reconstruction on Objaverse (unseen samples)](<https://www.sota2.com/research/sota/3d-reconstruction-on-objaverse-unseen-samples.md>): 3D Reconstruction; Objaverse (unseen samples); PSNR: 28.32.

- [Image-to-3D generation on Google Scanned Objects (GSO)](<https://www.sota2.com/research/sota/image-to-3d-generation-on-google-scanned-objects-gso.md>): Image-to-3D Generation; Google Scanned Objects (GSO); CLIP Similarity: 52.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2508.15228>)

[PDF](<https://arxiv.org/pdf/2508.15228.pdf>)
