# Compositional Foundation Models for Hierarchical Planning

Source: [Compositional Foundation Models for Hierarchical Planning](<https://www.sota2.com/research/paper/68fa70b51e209131cea4cb76>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:59:47.519Z

Authors: Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, Pulkit Agrawal

Published: 2023-09-15T17:44:05.000Z

## Abstract

To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-Horizon Robotics Manipulation on Paint-block (Seen)](<https://www.sota2.com/research/sota/long-horizon-robotics-manipulation-on-paint-block-seen.md>): Long-Horizon Robotics Manipulation; Paint-block (Seen); Success Rate: 81.2.

- [Long-Horizon Robotics Manipulation on Paint-block (Unseen)](<https://www.sota2.com/research/sota/long-horizon-robotics-manipulation-on-paint-block-unseen.md>): Long-Horizon Robotics Manipulation; Paint-block (Unseen); Success Rate: 79.6.

- [Long-Horizon Robotics Manipulation on Object-arrange (Seen)](<https://www.sota2.com/research/sota/long-horizon-robotics-manipulation-on-object-arrange-seen.md>): Long-Horizon Robotics Manipulation; Object-arrange (Seen); Success Rate: 91.8.

- [Long-Horizon Robotics Manipulation on Object-arrange (Unseen)](<https://www.sota2.com/research/sota/long-horizon-robotics-manipulation-on-object-arrange-unseen.md>): Long-Horizon Robotics Manipulation; Object-arrange (Unseen); Success Rate: 92.3.

- [Long-Horizon Robotics Manipulation on Kitchen-tasks (Seen)](<https://www.sota2.com/research/sota/long-horizon-robotics-manipulation-on-kitchen-tasks-seen.md>): Long-Horizon Robotics Manipulation; Kitchen-tasks (Seen); Success Rate: 0.928.

- [Long-Horizon Robotics Manipulation on Kitchen-tasks (Unseen)](<https://www.sota2.com/research/sota/long-horizon-robotics-manipulation-on-kitchen-tasks-unseen.md>): Long-Horizon Robotics Manipulation; Kitchen-tasks (Unseen); Success Rate: 89.8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2309.08587>)

[PDF](<https://arxiv.org/pdf/2309.08587.pdf>)

[Implementation](<https://hierarchical-planning-foundation-model.github.io/>)