# Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning

Source: [Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning](<https://www.sota2.com/research/paper/69ae53df2d603e6de3e428ae>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-10T05:00:05.424Z

Authors: Cristiano Battistini, Riccardo Andrea Izzo, Gianluca Bardaro, Matteo Matteucci

Published: 2026-03-06T09:36:29.000Z

## Abstract

Large and small language models have been widely used for robotic task planning. At the same time, vision-language models (VLMs) have successfully tackled problems such as image captioning, scene understanding, and visual question answering. In this work, we combine these two approaches by deploying a compact, open-source multimodal model to generate behavior trees for robotic task planning. The main obstacle to achieving this goal is the lack of an existing dataset that links visual observations and instructions to executable behavior trees. We propose a method to construct such a dataset starting from existing robotic episodes (i.e., Open X-Embodiment), in which a large model serves as a teacher in a multi-stage generation pipeline. We use this dataset to fine-tune VLMs ranging from 500M to 4B parameters via parameter-efficient fine-tuning (PEFT). The generated behavior trees, compatible with the BehaviorTree.CPP library, are evaluated both offline, using structural and lexical metrics, and online through the execution of household tasks in a state-of-the-art embodied simulator. Our results demonstrate that our fine-tuned 4B-parameter VLM approaches the performance of state-of-the-art closed-source models, achieving an 87\\% success rate while requiring only a fraction of the computational resources.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Simulation Task Planning on BEHAVIOR-1K 15 tasks](<https://www.sota2.com/research/sota/simulation-task-planning-on-behavior-1k-15-tasks.md>): Simulation Task Planning; BEHAVIOR-1K 15 tasks; BT Valid: 100.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.06084>)

[PDF](<https://arxiv.org/pdf/2603.06084.pdf>)
