Human Interaction-Aware 3D Reconstruction from a Single Image
About
Reconstructing textured 3D human models from a single image is fundamental for AR/VR and digital human applications. However, existing methods mostly focus on single individuals and thus fail in multi-human scenes, where naive composition of individual reconstructions often leads to artifacts such as unrealistic overlaps, missing geometry in occluded regions, and distorted interactions. These limitations highlight the need for approaches that incorporate group-level context and interaction priors. We introduce a holistic method that explicitly models both group- and instance-level information. To mitigate perspective-induced geometric distortions, we first transform the input into a canonical orthographic space. Our primary component, Human Group-Instance Multi-View Diffusion (HUG-MVD), then generates complete multi-view normals and images by jointly modeling individuals and group context to resolve occlusions and proximity. Subsequently, the Human Group-Instance Geometric Reconstruction (HUG-GR) module optimizes the geometry by leveraging explicit, physics-based interaction priors to enforce physical plausibility and accurately model inter-human contact. Finally, the multi-view images are fused into a high-fidelity texture. Together, these components form our complete framework, HUG3D. Extensive experiments show that HUG3D significantly outperforms both single-human and existing multi-human methods, producing physically plausible, high-fidelity 3D reconstructions of interacting people from a single image. Project page: https://jongheean11.github.io/HUG3D_project
Key insights
What problem does HUG3D solve?
Reconstructing textured 3D models of multiple interacting people from one image is hard due to occlusion, perspective distortion, and unrealistic overlaps. HUG3D transforms the image to a canonical orthographic space, uses a joint group-instance diffusion model to complete missing regions, and applies physics-based optimization for plausible contact.
How does it differ from prior work?
Unlike methods that reconstruct individuals independently or require multi-view input, HUG3D jointly models group and instance context. On the MultiHuman dataset, it achieves 3.631 cm Chamfer Distance and 0.240 contact precision, outperforming SIFU, SiTH, PSHuman, and DeepMultiCap across 80 two-person images.
What are the practical uses?
The authors target AR/VR, telepresence, and digital human modeling applications. The paper demonstrates a research prototype on benchmark and in-the-wild images but does not document any production deployment or commercial customers.
What limitations do the authors note?
HUG3D does not handle occlusions from external objects and can fail under severe depth ambiguity or heavy occlusion. It assumes ambient lighting, so low-light or high-contrast scenes may degrade. Errors in predicted SMPL-X pose can cause misaligned geometry, and no direct public baseline exists for this exact task.