# Probing the 3D Awareness of Visual Foundation Models

Source: [Probing the 3D Awareness of Visual Foundation Models](<https://www.sota2.com/research/paper/68fa72d81e209131cea58dbf>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T05:44:29.097Z

Authors: Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, Varun Jampani

Published: 2024-04-12T17:58:04.000Z

## Abstract

Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual backbone selection proxy evaluation on SimplerEnv WidowX + Bridge](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-simplerenv-widowx-plus-bridge.md>): Visual backbone selection proxy evaluation; SimplerEnv WidowX+Bridge; MMRV: 0.137.

- [Visual backbone selection proxy evaluation on MetaWorld](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-metaworld.md>): Visual backbone selection proxy evaluation; MetaWorld; MMRV: 0.109.

- [Visual backbone selection proxy evaluation on RoboCasa](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-robocasa.md>): Visual backbone selection proxy evaluation; RoboCasa; MMRV: 0.035.

- [Visual backbone selection proxy evaluation on SimplerEnv Google Robot](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-simplerenv-google-robot.md>): Visual backbone selection proxy evaluation; SimplerEnv Google Robot; MMRV: 0.101.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.08636>)

[PDF](<https://arxiv.org/pdf/2404.08636.pdf>)

[Implementation](<https://github.com/mbanani/probe3d>)
