# Capturing Visual Environment Structure Correlates with Control Performance

Source: [Capturing Visual Environment Structure Correlates with Control Performance](<https://www.sota2.com/research/paper/6984247ba8b01e76cc8eea10>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-06T05:00:14.101Z

Authors: Jiahua Dong, Yunze Man, Pavel Tokmakov, Yu-Xiong Wang

Published: 2026-02-04T18:59:12.000Z

## Abstract

The choice of visual representation is key to scaling generalist robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state -- including geometry, object structure, and physical attributes -- from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode the latent physical state of the environment is a promising objective for control.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual backbone selection proxy evaluation on SimplerEnv Google Robot](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-simplerenv-google-robot.md>): Visual backbone selection proxy evaluation; SimplerEnv Google Robot; MMRV: 0.023.

- [Visual backbone selection proxy evaluation on MetaWorld](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-metaworld.md>): Visual backbone selection proxy evaluation; MetaWorld; MMRV: 0.037.

- [Visual backbone selection proxy evaluation on RoboCasa](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-robocasa.md>): Visual backbone selection proxy evaluation; RoboCasa; MMRV: 0.01.

- [Visual backbone selection proxy evaluation on SimplerEnv WidowX + Bridge](<https://www.sota2.com/research/sota/visual-backbone-selection-proxy-evaluation-on-simplerenv-widowx-plus-bridge.md>): Visual backbone selection proxy evaluation; SimplerEnv WidowX+Bridge; MMRV: 0.069.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2602.04880>)

[PDF](<https://arxiv.org/pdf/2602.04880.pdf>)
