AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving
About
Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient fast-slow inference through asynchronous execution at different task frequencies. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves competitive performance compared to state-of-the-art methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when AD-tailored fine-tuning is necessary. Our results show that pre-trained VLMs can achieve competitive multi-task scene understanding performance through semantic prompting alone, while fine-tuning remains essential for action-level tasks such as decision-making and trajectory planning. We refer to https://automot-website.github.io/ for the demonstration videos and qualitative results.
Key insights
What problem does AutoMoT solve in autonomous driving?
AutoMoT combines scene reasoning and action generation in one driving model. Shared attention and cached representations let its reasoning component run more slowly than its action component, reducing the need to wait for fresh reasoning before every control step.
How does it differ from prior VLA driving approaches?
AutoMoT reports a CARLA Bench2Drive driving score of 87.34 versus SimLingo's 85.07. Its asynchronous setup reports 37 ms action latency, compared with 430 ms for SimLingo in the paper's latency comparison.
What are the practical uses of these findings?
The asynchronous design suits real-time autonomous driving needing both high-level reasoning and low-latency control. It is a benchmark prototype; the authors note onboard deployment remains resource-intensive but may become practical via acceleration.
What limitations and future work do the authors identify?
The authors note that stochastic diffusion policies can accumulate trajectory errors in closed-loop driving. Experiments cover simulation and open-loop benchmarks; the computational demands of onboard use remain an unresolved deployment constraint.