# StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning

Source: [StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning](<https://www.sota2.com/research/paper/69facb49c0f56ed84308a603>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-05T05:00:12.194Z

Authors: Xiaowen Sun, Matthias Kerzel, Mengdi Li, Xufeng Zhao, Paul Striker, Stefan Wermter

Published: 2026-05-05T16:19:02.000Z

## Abstract

Vision-language models (VLMs) have shown remarkable performance in various robotic tasks, as they can perceive visual information and understand natural language instructions. However, when applied to robotics, VLMs remain subject to a fundamental limitation inherent in large language models (LLMs): they struggle with numerical reasoning, particularly in object detection and object-state localization. To explore numerical reasoning as a regression task in VLMs, we propose a novel training strategy to adapt VLMs for object detection and object-state localization. This approach leverages box decoder outputs to compute an Auxiliary Regression Loss (ARL) during fine-tuning, while preserving standard sequence prediction at inference. We leverage this training strategy to develop StateVLM (State-aware Vision-Language Model), a novel model designed to perceive and learn fine-grained object representations, including precise localization of objects and their states, as well as graspable regions. Due to the lack of a benchmark for object-state affordance reasoning, we introduce an open-source benchmark, Object State Affordance Reasoning (OSAR), which contains 1172 scenes with 7746 individual objects and corresponding bounding boxes. Comparative experiments on adapted benchmarks (RefCOCO, RefCOCO+, and RefCOCOg) demonstrate that ARL improves model performance by an average of 1.6% compared to models without ARL. Experiments on the OSAR benchmark further support this finding, showing that StateVLM with ARL achieves an average of 5.2% higher performance than models without ARL. In particular, ARL is also important for the complex task of affordance reasoning in OSAR, where it enhances the consistency of model outputs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Referring Expression Comprehension on RefCOCO+ (val)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcoco-plus-val.md>): Referring Expression Comprehension; RefCOCO+ (val); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCO (testA)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcoco-testa.md>): Referring Expression Comprehension; RefCOCO (testA); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCO (val)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcoco-val.md>): Referring Expression Comprehension; RefCOCO (val); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCOg (val)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcocog-val.md>): Referring Expression Comprehension; RefCOCOg (val); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCOg (test)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcocog-test.md>): Referring Expression Comprehension; RefCOCOg (test); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCO+ (testB)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcoco-plus-testb.md>): Referring Expression Comprehension; RefCOCO+ (testB); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCO+ (testA)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcoco-plus-testa.md>): Referring Expression Comprehension; RefCOCO+ (testA); Accuracy: not recorded.

- [Referring Expression Comprehension on RefCOCO (testB)](<https://www.sota2.com/research/sota/referring-expression-comprehension-on-refcoco-testb.md>): Referring Expression Comprehension; RefCOCO (testB); Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.03927>)

[PDF](<https://arxiv.org/pdf/2605.03927.pdf>)
