# Instruction-following robotic manipulation on CALVIN ABC→D (unseen environment D)

Source: [Instruction-following robotic manipulation on CALVIN ABC→D (unseen environment D)](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:05:38.432Z

Evaluation of instruction-following robotic manipulation performance on the CALVIN ABC→D benchmark, specifically focusing on the unseen environment D split. Metrics include success rates conditioned on the length of the instruction sequence (1 to 5 steps) and average trajectory length.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Instruction-following robotic manipulation |
| Dataset (including any stated split) | CALVIN ABC→D (unseen environment D) |
| Metric | Success Rate (Length 1) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 29 of 29 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Success Rate (Length 1)](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d.md?metric=Success%20Rate%20(Length%201)>): Higher is better; unit: PERCENT.

- [Success Rate (Length 2)](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d.md?metric=Success%20Rate%20(Length%202)>): Higher is better; unit: PERCENT.

- [Success Rate (Length 3)](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d.md?metric=Success%20Rate%20(Length%203)>): Higher is better; unit: PERCENT.

- [Success Rate (Length 4)](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d.md?metric=Success%20Rate%20(Length%204)>): Higher is better; unit: PERCENT.

- [Success Rate (Length 5)](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d.md?metric=Success%20Rate%20(Length%205)>): Higher is better; unit: PERCENT.

- [Avg Trajectory Length](<https://www.sota2.com/research/sota/instruction-following-robotic-manipulation-on-calvin-abc-d-unseen-environment-d.md?metric=Avg%20Trajectory%20Length>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Success Rate (Length 1) | Result date | Report paper |
| --- | --- | --- | --- |
| HiF-VLA | 98.5 | Not recorded | 693a50316cfd488761a54c65 |
| RoboVLMs | 98 | Not recorded | 693a50316cfd488761a54c65 |
| PALM | 96.9 | Not recorded | 6965d1ae6343fa78c832389c |
| VPP | 96.5 | Not recorded | 693a50316cfd488761a54c65 |
| OpenVLA-OFT | 96.3 | Not recorded | 693a50316cfd488761a54c65 |
| Seer | 96.3 | Not recorded | 693a50316cfd488761a54c65 |
| CLOVER | 96 | Not recorded | 693a50316cfd488761a54c65 |
| UniVLA | 95.5 | Not recorded | 693a50316cfd488761a54c65 |
| PALM (✗ progress) | 95.3 | Not recorded | 6965d1ae6343fa78c832389c |
| Seer | 94.4 | Not recorded | 6965d1ae6343fa78c832389c |
| RoboUniview | 94.2 | Not recorded | 6965d1ae6343fa78c832389c |
| π0 | 93.8 | Not recorded | 6965d1ae6343fa78c832389c |
| π0 | 93.8 | Not recorded | 693a50316cfd488761a54c65 |
| π0 | 93.7 | Not recorded | 693a50316cfd488761a54c65 |
| HiF-VLA | 93.5 | Not recorded | 693a50316cfd488761a54c65 |
| UP-VLA | 92.8 | Not recorded | 693a50316cfd488761a54c65 |
| 3D Diffuser Actor | 92.2 | Not recorded | 6965d1ae6343fa78c832389c |
| Vidman | 91.5 | Not recorded | 693a50316cfd488761a54c65 |
| OpenVLA | 91.3 | Not recorded | 6965d1ae6343fa78c832389c |
| OpenVLA | 91.3 | Not recorded | 693a50316cfd488761a54c65 |
| VPP | 90.9 | Not recorded | 693a50316cfd488761a54c65 |
| Susie | 87 | Not recorded | 6965d1ae6343fa78c832389c |
| SuSIE | 87 | Not recorded | 693a50316cfd488761a54c65 |
| GR-1 | 85.4 | Not recorded | 6965d1ae6343fa78c832389c |
| GR-1 | 85.4 | Not recorded | 693a50316cfd488761a54c65 |
| Robo-Flamingo | 82.4 | Not recorded | 6965d1ae6343fa78c832389c |
| RT-1 | 53.3 | Not recorded | 6965d1ae6343fa78c832389c |
| 3D-VLA | 44.7 | Not recorded | 6965d1ae6343fa78c832389c |
| Diffusion Policy | 40.2 | Not recorded | 6965d1ae6343fa78c832389c |

### HiF-VLA

Success Rate (Length 1): 98.5. Result ID: 6999eec65698665ec46f868f.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Implementation](<https://github.com/OpenHelix-Team/HiF-VLA>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### RoboVLMs

Success Rate (Length 1): 98. Result ID: 6999eec65698665ec46f868c.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76601e209131cea6cbc6.md>)

[Implementation](<https://www.sota2.com/robovlms.github.io>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### PALM

Success Rate (Length 1): 96.9. Result ID: 6999daa55698665ec459cf89.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

Recorded evaluation configuration: {"Type":"Prediction + Progress"}. This record does not assert independent reproduction.

### VPP

Success Rate (Length 1): 96.5. Result ID: 6999eec65698665ec46f868e.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78581e209131cea77da7.md>)

[Implementation](<https://video-prediction-policy.github.io/>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### OpenVLA-OFT

Success Rate (Length 1): 96.3. Result ID: 6999eec65698665ec46f868b.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78471e209131cea7794b.md>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### Seer

Success Rate (Length 1): 96.3. Result ID: 6999eec65698665ec46f868d.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa764a1e209131cea6c2b0.md>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### CLOVER

Success Rate (Length 1): 96. Result ID: 6999eec65698665ec46f8682.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa753e1e209131cea66ba3.md>)

[Implementation](<https://github.com/OpenDriveLab/CLOVER>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### UniVLA

Success Rate (Length 1): 95.5. Result ID: 6999eec65698665ec46f8685.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78851e209131cea78d36.md>)

[Implementation](<https://github.com/OpenDriveLab/UniVLA>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### PALM (✗ progress)

Success Rate (Length 1): 95.3. Result ID: 6999daa55698665ec459cf88.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

Recorded evaluation configuration: {"Type":"Prediction"}. This record does not assert independent reproduction.

### Seer

Success Rate (Length 1): 94.4. Result ID: 6999daa55698665ec459cf87.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa764a1e209131cea6c2b0.md>)

Recorded evaluation configuration: {"Type":"Prediction"}. This record does not assert independent reproduction.

### RoboUniview

Success Rate (Length 1): 94.2. Result ID: 6999daa55698665ec459cf84.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74cc1e209131cea63eb1.md>)

[Implementation](<https://github.com/liufanfanlff/RoboUniview>)

Recorded evaluation configuration: {"Type":"3D-Aware"}. This record does not assert independent reproduction.

### π0

Success Rate (Length 1): 93.8. Result ID: 6999daa55698665ec459cf81.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75aa1e209131cea6929a.md>)

Recorded evaluation configuration: {"Type":"Diffusion-based"}. This record does not assert independent reproduction.

### π0

Success Rate (Length 1): 93.8. Result ID: 6999eec65698665ec46f8689.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75aa1e209131cea6929a.md>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### π0

Success Rate (Length 1): 93.7. Result ID: 6999eec65698665ec46f8684.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75aa1e209131cea6929a.md>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### HiF-VLA

Success Rate (Length 1): 93.5. Result ID: 6999eec65698665ec46f8686.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Implementation](<https://github.com/OpenHelix-Team/HiF-VLA>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### UP-VLA

Success Rate (Length 1): 92.8. Result ID: 6999eec65698665ec46f868a.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79a11e209131cea7e8d4.md>)

[Implementation](<https://github.com/CladernyJorn/UP-VLA>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### 3D Diffuser Actor

Success Rate (Length 1): 92.2. Result ID: 6999daa55698665ec459cf83.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa744b1e209131cea60daf.md>)

[Implementation](<https://www.sota2.com/3d-diffuser-actor.github.io>)

Recorded evaluation configuration: {"Type":"3D-Aware"}. This record does not assert independent reproduction.

### Vidman

Success Rate (Length 1): 91.5. Result ID: 6999eec65698665ec46f8688.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75af1e209131cea69482.md>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### OpenVLA

Success Rate (Length 1): 91.3. Result ID: 6999daa55698665ec459cf7f.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74c31e209131cea63a36.md>)

[Implementation](<https://openvla.github.io/>)

Recorded evaluation configuration: {"Type":"Autoregressive"}. This record does not assert independent reproduction.

### OpenVLA

Success Rate (Length 1): 91.3. Result ID: 6999eec65698665ec46f8681.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74c31e209131cea63a36.md>)

[Implementation](<https://openvla.github.io/>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### VPP

Success Rate (Length 1): 90.9. Result ID: 6999eec65698665ec46f8683.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78581e209131cea77da7.md>)

[Implementation](<https://video-prediction-policy.github.io/>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### Susie

Success Rate (Length 1): 87. Result ID: 6999daa55698665ec459cf85.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70ec1e209131cea4e08c.md>)

Recorded evaluation configuration: {"Type":"Prediction"}. This record does not assert independent reproduction.

### SuSIE

Success Rate (Length 1): 87. Result ID: 6999eec65698665ec46f8680.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70ec1e209131cea4e08c.md>)

Recorded evaluation configuration: {"View":"Third View"}. This record does not assert independent reproduction.

### GR-1

Success Rate (Length 1): 85.4. Result ID: 6999daa55698665ec459cf86.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71861e209131cea51c7f.md>)

[Implementation](<https://gr1-manipulation.github.io/>)

Recorded evaluation configuration: {"Type":"Prediction"}. This record does not assert independent reproduction.

### GR-1

Success Rate (Length 1): 85.4. Result ID: 6999eec65698665ec46f8687.

[Reporting paper](<https://www.sota2.com/research/paper/693a50316cfd488761a54c65.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71861e209131cea51c7f.md>)

[Implementation](<https://gr1-manipulation.github.io/>)

Recorded evaluation configuration: {"View":"Multi-View"}. This record does not assert independent reproduction.

### Robo-Flamingo

Success Rate (Length 1): 82.4. Result ID: 6999daa55698665ec459cf7e.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa721c1e209131cea548b5.md>)

Recorded evaluation configuration: {"Type":"Autoregressive"}. This record does not assert independent reproduction.

### RT-1

Success Rate (Length 1): 53.3. Result ID: 6999daa55698665ec459cf7d.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70411e209131cea49f7d.md>)

[Implementation](<http://github.com/google-research/robotics_transformer>)

Recorded evaluation configuration: {"Type":"Autoregressive"}. This record does not assert independent reproduction.

### 3D-VLA

Success Rate (Length 1): 44.7. Result ID: 6999daa55698665ec459cf82.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa727e1e209131cea56bda.md>)

Recorded evaluation configuration: {"Type":"3D-Aware"}. This record does not assert independent reproduction.

### Diffusion Policy

Success Rate (Length 1): 40.2. Result ID: 6999daa55698665ec459cf80.

[Reporting paper](<https://www.sota2.com/research/paper/6965d1ae6343fa78c832389c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72841e209131cea56f68.md>)

Recorded evaluation configuration: {"Type":"Diffusion-based"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.