OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
About
Since the advent of Multimodal Large Language Models (MLLMs), they have made a significant impact across a wide range of real-world applications, particularly in Autonomous Driving (AD). Their ability to process complex visual data and reason about intricate driving scenarios has paved the way for a new paradigm in end-to-end AD systems. However, the progress of developing end-to-end models for AD has been slow, as existing fine-tuning methods demand substantial resources, including extensive computational power, large-scale datasets, and significant funding. Drawing inspiration from recent advancements in inference computing, we propose OpenEMMA, an open-source end-to-end framework based on MLLMs. By incorporating the Chain-of-Thought reasoning process, OpenEMMA achieves significant improvements compared to the baseline when leveraging a diverse range of MLLMs. Furthermore, OpenEMMA demonstrates effectiveness, generalizability, and robustness across a variety of challenging driving scenarios, offering a more efficient and effective approach to autonomous driving. We release all the codes in https://github.com/taco-group/OpenEMMA.
Key insights
What problem does OpenEMMA address and how?
It replicates Waymo's closed-source EMMA using open MLLMs for end-to-end autonomous driving. Front-camera images and ego-vehicle history drive chain-of-thought reasoning that predicts speed and curvature, integrated into waypoints; a fine-tuned YOLO3D handles 3D detection.
How does it differ from prior work?
Unlike fine-tuned approaches like EMMA, OpenEMMA uses off-the-shelf MLLMs without fine-tuning. On 150 nuScenes validation scenes, OpenEMMA with LLaVA-1.6-Mistral-7B achieves 2.98m average L2 error versus 3.24m for the zero-shot baseline.
What are the practical uses?
OpenEMMA provides a reproducible platform for driving research: researchers can inspect its reasoning, trajectory planning and external 3D detector. The reported experiments use nuScenes data, so they establish benchmark behavior rather than on-road autonomy.
What limitations do the authors note?
OpenEMMA uses only basic chain-of-thought reasoning, with untapped potential for advanced techniques like CoT-SC and ToT. Current MLLMs lack object grounding capability, requiring an external YOLO model rather than a fully unified MLLM framework.