# Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

Source: [Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling](<https://www.sota2.com/research/paper/698abbc4c97a0b3e3b5de121>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-30T05:00:15.133Z

Authors: Xihang Yu, Rajat Talak, Lorenzo Shaikewitz, Luca Carlone

Published: 2026-02-08T17:04:54.000Z

## Abstract

In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions -- which fit the sensor data -- can still be physically incorrect. For instance, when estimating the poses and shapes of objects in the scene and importing the resulting estimates into a simulator, small errors might translate to implausible configurations including object interpenetration or unstable equilibrium. This makes it difficult to predict the dynamic behavior of the scene using a digital twin, an important step in simulation-based planning and control of contact-rich behaviors. In this paper, we posit that object pose and shape estimation requires reasoning holistically over the scene (instead of reasoning about each object in isolation), accounting for object interactions and physical plausibility. Towards this goal, our first contribution is Picasso, a physics-constrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics. Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples. Second, we propose the Picasso dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our benchmark. Finally, we provide an extensive evaluation of Picasso on our newly introduced dataset and on the YCB-V dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [6D Object Pose Estimation on YCB-V BOP19 challenge (test)](<https://www.sota2.com/research/sota/6d-object-pose-estimation-on-ycb-v-bop19-challenge-test.md>): 6D Object Pose Estimation; YCB-V BOP19 challenge (test); ADD-S Mean: 0.008.

- [Pose Estimation on Picasso 1.0 (Overall)](<https://www.sota2.com/research/sota/pose-estimation-on-picasso-1-0-overall.md>): Pose Estimation; Picasso 1.0 (Overall); ADD-S: 7.55.

- [Physics Plausibility Evaluation on YCB-Video 12 trajectories (Overall)](<https://www.sota2.com/research/sota/physics-plausibility-evaluation-on-ycb-video-12-trajectories-overall.md>): Physics Plausibility Evaluation; YCB-Video 12 trajectories (Overall); SPS: 1.51.

- [6D Pose Estimation on YCB-V (SPS/NPS/OC)](<https://www.sota2.com/research/sota/6d-pose-estimation-on-ycb-v-sps-nps-oc.md>): 6D Pose Estimation; YCB-V; SPS: 8.77.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2602.08058>)

[PDF](<https://arxiv.org/pdf/2602.08058.pdf>)
