# Robotic Visual Instruction

Source: [Robotic Visual Instruction](<https://www.sota2.com/research/paper/68fa7a311e209131cea81aa4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:27:23.764Z

Authors: Yanbang Li, Ziyang Gong, Haoyang Li, Xiaoqi Huang, Haolan Kang, Guangping Bai, Xianzheng Ma

Published: 2025-05-01T17:55:05.000Z

## Abstract

Recently, natural language has been the primary medium for human-robot interaction. However, its inherent lack of spatial precision introduces challenges for robotic task definition such as ambiguity and verbosity. Moreover, in some public settings where quiet is required, such as libraries or hospitals, verbal communication with robots is inappropriate. To address these limitations, we introduce the Robotic Visual Instruction (RoVI), a novel paradigm to guide robotic tasks through an object-centric, hand-drawn symbolic representation. RoVI effectively encodes spatial-temporal information into human-interpretable visual instructions through 2D sketches, utilizing arrows, circles, colors, and numbers to direct 3D robotic manipulation. To enable robots to understand RoVI better and generate precise actions based on RoVI, we present Visual Instruction Embodied Workflow (VIEW), a pipeline formulated for RoVI-conditioned policies. This approach leverages Vision-Language Models (VLMs) to interpret RoVI inputs, decode spatial and temporal constraints from 2D pixel space via keypoint extraction, and then transform them into executable 3D action sequences. We additionally curate a specialized dataset of 15K instances to fine-tune small VLMs for edge deployment,enabling them to effectively learn RoVI capabilities. Our approach is rigorously validated across 11 novel tasks in both real and simulated environments, demonstrating significant generalization capability. Notably, VIEW achieves an 87.5% success rate in real-world scenarios involving unseen tasks that feature multi-step actions, with disturbances, and trajectory-following requirements. Project website: https://robotic-visual-instruction.github.io/

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Robotic Manipulation on Real World (unseen environments and tasks)](<https://www.sota2.com/research/sota/robotic-manipulation-on-real-world-unseen-environments-and-tasks.md>): Robotic Manipulation; Real World (unseen environments and tasks); Task 1 Success Rate: 90.

- [Robotic Manipulation on SIMPLER Simulator unseen tasks](<https://www.sota2.com/research/sota/robotic-manipulation-on-simpler-simulator-unseen-tasks.md>): Robotic Manipulation; SIMPLER Simulator unseen tasks; Task 1 Success Rate: 70.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.00693>)

[PDF](<https://arxiv.org/pdf/2505.00693.pdf>)

[Implementation](<https://robotic-visual-instruction.github.io/>)