# Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap

Source: [Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap](<https://www.sota2.com/research/paper/68fa6ce71e209131cea36a75>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:21:36.840Z

Authors: Martina Lippi, Petra Poklukar, Michael C. Welle, Anastasia Varava, Hang Yin, Alessandro Marino, Danica Kragic

Published: 2021-03-03T17:48:26.000Z

## Abstract

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a Latent Space Roadmap (LSR) for task planning which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of three parts: (1) a Mapping Module (MM) that maps observations given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states, (2) the LSR which builds and connects clusters containing similar states in order to find the latent plans between start and goal states extracted by MM, and (3) the Action Proposal Module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Task Planning and Graph Learning on Fruit-2x3](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-fruit-2x3.md>): Visual Task Planning and Graph Learning; Fruit-2x3; Optimality Score: 100.

- [Visual Task Planning and Graph Learning on Multi Fruit 2x3](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-multi-fruit-2x3.md>): Visual Task Planning and Graph Learning; Multi Fruit 2x3; Opt: 40.

- [Visual Task Planning and Graph Learning on Blocks 2](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-blocks-2.md>): Visual Task Planning and Graph Learning; Blocks 2; Opt: 76.7.

- [Visual Task Planning and Graph Learning on Blocks 3](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-blocks-3.md>): Visual Task Planning and Graph Learning; Blocks 3; Opt: 33.6.

- [Visual Task Planning and Graph Learning on Fruit-4x6](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-fruit-4x6.md>): Visual Task Planning and Graph Learning; Fruit-4x6; Opt Rate: 37.5.

- [Visual Task Planning and Graph Learning on Fruit 6x8](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-fruit-6x8.md>): Visual Task Planning and Graph Learning; Fruit 6x8; Opt: 52.7.

- [Visual Task Planning and Graph Learning on Multi Fruit 4x6](<https://www.sota2.com/research/sota/visual-task-planning-and-graph-learning-on-multi-fruit-4x6.md>): Visual Task Planning and Graph Learning; Multi Fruit 4x6; Opt: 12.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2103.02554>)

[PDF](<https://arxiv.org/pdf/2103.02554.pdf>)