# CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models

Source: [CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models](<https://www.sota2.com/research/paper/68fa72831e209131cea56ecc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:27:53.383Z

Authors: Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, Yang Gao

Published: 2024-03-13T05:03:58.000Z

## Abstract

Foundation models pre-trained on web-scale data are shown to encapsulate extensive world knowledge beneficial for robotic manipulation in the form of task planning. However, the actual physical implementation of these plans often relies on task-specific learning methods, which require significant data collection and struggle with generalizability. In this work, we introduce Robotic Manipulation through Spatial Constraints of Parts (CoPa), a novel framework that leverages the common sense knowledge embedded within foundation models to generate a sequence of 6-DoF end-effector poses for open-world robotic manipulation. Specifically, we decompose the manipulation process into two phases: task-oriented grasping and task-aware motion planning. In the task-oriented grasping phase, we employ foundation vision-language models (VLMs) to select the object's grasping part through a novel coarse-to-fine grounding mechanism. During the task-aware motion planning phase, VLMs are utilized again to identify the spatial geometry constraints of task-relevant object parts, which are then used to derive post-grasp poses. We also demonstrate how CoPa can be seamlessly integrated with existing robotic planning algorithms to accomplish complex, long-horizon tasks. Our comprehensive real-world experiments show that CoPa possesses a fine-grained physical understanding of scenes, capable of handling open-set instructions and objects with minimal prompt engineering and without additional training. Project page: https://copa-2024.github.io/

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Articulated-object manipulation on Real-world 3 articulated-object tasks](<https://www.sota2.com/research/sota/articulated-object-manipulation-on-real-world-3-articulated-object-tasks.md>): Articulated Object Manipulation; Real-world 3 articulated-object tasks; Drawer Opening Success Rate: 40.

- [Average Robotic Manipulation Success on Real-world Hardware](<https://www.sota2.com/research/sota/average-robotic-manipulation-success-on-real-world-hardware.md>): Average Robotic Manipulation Success; Real-world Hardware; Success Rate: 60.

- [Hammer nail on Real-world Hardware](<https://www.sota2.com/research/sota/hammer-nail-on-real-world-hardware.md>): Hammer Nail; Real-world Hardware; Success Rate: 30.

- [Knock tower on Real-world Hardware](<https://www.sota2.com/research/sota/knock-tower-on-real-world-hardware.md>): Knock tower; Real-world Hardware; Success Rate: 80.

- [Reach blocks on Real-world Hardware](<https://www.sota2.com/research/sota/reach-blocks-on-real-world-hardware.md>): Reach blocks; Real-world Hardware; Success Rate: 60.

- [Sweep toys on Real-world Hardware](<https://www.sota2.com/research/sota/sweep-toys-on-real-world-hardware.md>): Sweep toys; Real-world Hardware; Success Rate: 70.

- [Robotic Manipulation on Real World (unseen environments and tasks)](<https://www.sota2.com/research/sota/robotic-manipulation-on-real-world-unseen-environments-and-tasks.md>): Robotic Manipulation; Real World (unseen environments and tasks); Task 1 Success Rate: 40.

- [Object-centric manipulation on Real-world 10 object-centric tasks (Success Rate)](<https://www.sota2.com/research/sota/object-centric-manipulation-on-real-world-10-object-centric-tasks-success-rate.md>): Object-centric manipulation; Real-world 10 object-centric tasks; Egg Placing Success Rate: 20.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.08248>)

[PDF](<https://arxiv.org/pdf/2403.08248.pdf>)