# A General Protocol to Probe Large Vision Models for 3D Physical Understanding Source: [A General Protocol to Probe Large Vision Models for 3D Physical Understanding]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:37:22.778Z Authors: Guanqi Zhan, Chuanxia Zheng, Weidi Xie, Andrew Zisserman Published: 2023-10-10T17:59:28.000Z ## Abstract Our objective in this paper is to probe large vision models to determine to what extent they 'understand' different physical properties of the 3D scene depicted in an image. To this end, we make the following contributions: (i) We introduce a general and lightweight protocol to evaluate whether features of an off-the-shelf large vision model encode a number of physical 'properties' of the 3D scene, by training discriminative classifiers on the features for these properties. The probes are applied on datasets of real images with annotations for the property. (ii) We apply this protocol to properties covering scene geometry, scene material, support relations, lighting, and view-dependent measures, and large vision models including CLIP, DINOv1, DINOv2, VQGAN, Stable Diffusion. (iii) We find that features from Stable Diffusion and DINOv2 are good for discriminative learning of a number of properties, including scene geometry, support relations, shadows and depth, but less performant for occlusion and material, while outperforming DINOv1, CLIP and VQGAN for all properties. (iv) It is observed that different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, are good at different properties, unlocking potential applications of 3D physical understanding. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Depth classification on 3D physical scene properties (test)](): Depth classification; 3D physical scene properties (test); ROC AUC: not recorded. - [Material classification on 3D physical scene properties (test)](): Material Classification; 3D physical scene properties (test); ROC AUC: not recorded. - [Occlusion detection on 3D physical scene properties (test)](): Occlusion Detection; 3D physical scene properties (test); ROC AUC: not recorded. - [Perpendicular Plane prediction on 3D physical scene properties (test)](): Perpendicular Plane prediction; 3D physical scene properties (test); ROC AUC: not recorded. - [Same Plane prediction on 3D physical scene properties (test)](): Same Plane prediction; 3D physical scene properties (test); ROC AUC: not recorded. - [Shadow detection on 3D physical scene properties (test)](): Shadow Detection; 3D physical scene properties (test); ROC AUC: not recorded. - [Support Relation prediction on 3D physical scene properties (test)](): Support Relation prediction; 3D physical scene properties (test); ROC AUC: not recorded. ## Original sources and implementation [Original paper]() [PDF]()