# GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations Source: [GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations]() Collection index: [llms.txt]() Data updated: 2026-05-23T05:00:11.413Z Authors: Wenxuan Guo, Ziyuan Li, Meng Zhang, Yichen Liu, Yimeng Dong, Chuxi Xu, Yunfei Wei, Ze Chen, Erjin Zhou, Jianjiang Feng Published: 2026-05-21T17:57:44.000Z ## Abstract Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action. However, existing VLA systems primarily rely on textual instructions and struggle to resolve spatial ambiguity in complex scenes with multiple similar objects. To address this limitation, we introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA). Our approach encodes gesture features directly into the latent space, enabling them to participate in both high-level reasoning and low-level action generation, and adopts a dual-VLM architecture to achieve tight coupling between gesture representations and action policies. At the data level, we construct a scalable gesture data generation pipeline by rendering hand models onto real-world scene images. This reduces the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations. In addition, we employ a two-stage training strategy to equip the model with both gesture perception and action prediction capabilities. We evaluate our approach on multiple real-world robotic tasks, including a controlled block manipulation task for validation and more practical scenarios such as product and produce selection. Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency, especially in complex and cluttered environments. Project page: https://gwxuan.github.io/GesVLA/. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Pick-and-place block on Pick-and-Place Block Simple](): Pick & Place Block; Pick-and-Place Block Simple; Success Rate: 100. - [Pick-and-place block on Pick-and-Place Block Hard](): Pick & Place Block; Pick-and-Place Block Hard; Success Rate: 90. - [Select fruit and vegetable on Select Fruit and Vegetable Simple](): Select fruit and vegetable; Select Fruit and Vegetable Simple; Success Rate: 8. - [Select fruit and vegetable on Select Fruit and Vegetable Hard](): Select fruit and vegetable; Select Fruit and Vegetable Hard; Success Rate: 80. - [Select jelly on Select Jelly Simple](): Select jelly; Select Jelly Simple; Success Rate: 90. - [Select jelly on Select Jelly Hard](): Select jelly; Select Jelly Hard; Success Rate: 60. - [Intent reasoning on Real-world intent reasoning 88 samples (test)](): Intent reasoning; Real-world intent reasoning 88 samples (test); Accuracy: 94.3. ## Original sources and implementation [Original paper]() [PDF]()