Robotic Manipulation Task Code Generation on Tabletop Manipulation simulator
100Execution Success RateLang2Code
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Lang2CodeTask=Place A next to B, Category=Specificity2023.05 | 100 | 33 | 0.73 | |
| Lang2CodeTask=Place A at a corner of the table, Category=Specificity2023.05 | 100 | 30 | 0.08 | |
| Lang2CodeTask=Place A at an edge of the table, Category=Specificity2023.05 | 100 | 20 | 0.59 | |
| Lang2CodeTask=Place A on top of B, Category=Hidden World Constraint2023.05 | 100 | 3 | 0.23 | |
| Lang2CodeTask=Stack all blocks, Category=Hidden World Constraint2023.05 | 100 | 20 | 0.87 | |
| Lang2CodeTask=Stack all cylinders, Category=Hidden World Constraint2023.05 | 100 | 37 | 0.89 | |
| Lang2CodeTask=Stack all blocks into one stack, Category=Personal Preference2023.05 | 100 | 13 | 0.07 | |
| Lang2CodeTask=Stack all cylinders into one stack, Category=Personal Preference2023.05 | 100 | 13 | 0 | |
| Lang2CodeTask=Stack all objects into two stacks, Category=Personal Preference2023.05 | 100 | 0 | 0 | |
| Lang2CodeTask=Overall2023.05 | 100 | 19 | 0.39 | |
| DemoNoLang2CodeTask=Place A at a corner of the table, Category=Specificity2023.05 | 100 | 100 | 0.83 | |
| DemoNoLang2CodeTask=Place A at an edge of the table, Category=Specificity2023.05 | 100 | 95 | 0.84 | |
| DemoNoLang2CodeTask=Stack all blocks, Category=Hidden World Constraint2023.05 | 100 | 70 | 0.5 | |
| DemoNoLang2CodeTask=Stack all cylinders, Category=Hidden World Constraint2023.05 | 100 | 83 | 0.49 | |
| DemoNoLang2CodeTask=Stack all blocks into one stack, Category=Personal Preference2023.05 | 100 | 67 | 0.52 | |
| DemoNoLang2CodeTask=Stack all objects into two stacks, Category=Personal Preference2023.05 | 100 | 90 | 0.68 | |
| Demo2CodeTask=Place A next to B, Category=Specificity2023.05 | 100 | 100 | 0.98 | |
| Demo2CodeTask=Place A at a corner of the table, Category=Specificity2023.05 | 100 | 100 | 1 | |
| Demo2CodeTask=Place A at an edge of the table, Category=Specificity2023.05 | 100 | 100 | 0.84 | |
| Demo2CodeTask=Stack all blocks, Category=Hidden World Constraint2023.05 | 100 | 70 | 0.5 | |
| Demo2CodeTask=Stack all cylinders, Category=Hidden World Constraint2023.05 | 100 | 100 | 1 | |
| Demo2CodeTask=Stack all blocks into one stack, Category=Personal Preference2023.05 | 100 | 87 | 0.71 | |
| Demo2CodeTask=Stack all cylinders into one stack, Category=Personal Preference2023.05 | 100 | 90 | 0.58 | |
| Demo2CodeTask=Stack all objects into two stacks, Category=Personal Preference2023.05 | 100 | 90 | 0.65 | |
| Demo2CodeTask=Overall2023.05 | 99 | 88 | 0.77 | |
| DemoNoLang2CodeTask=Overall2023.05 | 93 | 81 | 0.6 | |
| DemoNoLang2CodeTask=Place A next to B, Category=Specificity2023.05 | 90 | 80 | 0.82 | |
| DemoNoLang2CodeTask=Stack all cylinders into one stack, Category=Personal Preference2023.05 | 90 | 77 | 0.19 | |
| Demo2CodeTask=Place A on top of B, Category=Hidden World Constraint2023.05 | 90 | 40 | 0.4 | |
| DemoNoLang2CodeTask=Place A on top of B, Category=Hidden World Constraint2023.05 | 60 | 70 | 0.56 |