# A Vision-Language-Action Model with Visual Prompt for OFF-Road Autonomous Driving

Source: [A Vision-Language-Action Model with Visual Prompt for OFF-Road Autonomous Driving](<https://www.sota2.com/research/paper/695f39e453fdd5c361743d3b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-14T05:00:12.520Z

Authors: Liangdong Zhang, Yiming Nie, Haoyang Li, Fanjie Kong, Baobao Zhang, Shunxin Huang, Kai Fu, Chen Min, Liang Xiao

Published: 2026-01-07T02:08:18.000Z

## Abstract

Efficient trajectory planning in off-road terrains presents a formidable challenge for autonomous vehicles, often necessitating complex multi-step pipelines. However, traditional approaches exhibit limited adaptability in dynamic environments. To address these limitations, this paper proposes OFF-EMMA, a novel end-to-end multimodal framework designed to overcome the deficiencies of insufficient spatial perception and unstable reasoning in visual-language-action (VLA) models for off-road autonomous driving scenarios. The framework explicitly annotates input images through the design of a visual prompt block and introduces a chain-of-thought with self-consistency (COT-SC) reasoning strategy to enhance the accuracy and robustness of trajectory planning. The visual prompt block utilizes semantic segmentation masks as visual prompts, enhancing the spatial understanding ability of pre-trained visual-language models for complex terrains. The COT- SC strategy effectively mitigates the error impact of outliers on planning performance through a multi-path reasoning mechanism. Experimental results on the RELLIS-3D off-road dataset demonstrate that OFF-EMMA significantly outperforms existing methods, reducing the average L2 error of the Qwen backbone model by 13.3% and decreasing the failure rate from 16.52% to 6.56%.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Trajectory Prediction on RELLIS-3D](<https://www.sota2.com/research/sota/trajectory-prediction-on-rellis-3d.md>): Trajectory Prediction; RELLIS-3D; L2 Error (1s): 0.88.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.03519>)

[PDF](<https://arxiv.org/pdf/2601.03519.pdf>)
