Robotic Manipulation on CALVIN ABCD->D (Success Rate conditioned on Instruction Length)
0.4Avg LengthMCIL
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| MCILTask=ABCD→D2026.03 | 0.4 | 37.3 | 2.7 | 0.2 | 0 | 0 | |
| Diffusion PolicySize=–2026.04 | 0.56 | 40.2 | 12.3 | 2.6 | 0.8 | 0 | |
| RT-12026.01 | 0.9 | 53.3 | 22.2 | 9.4 | 3.8 | 1.3 | |
| UniPiArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=pixel-level2026.03 | 0.92 | 56 | 16 | 8 | 8 | 4 | |
| DiffPolicy2026.01 | 1.35 | 63.5 | 35.3 | 19.4 | 10.7 | 6.4 | |
| Diffusion PolicySampling=Original2026.05 | 2.41 | 82.1 | 61.7 | 45.6 | 31.4 | 20.5 | |
| RT-1Task=ABCD→D2026.03 | 2.45 | 84.4 | 61.7 | 43.8 | 32.3 | 22.7 | |
| RoboFlamingo2026.01 | 2.47 | 82.4 | 61.9 | 46.6 | 33.1 | 23.5 | |
| Robo-Flamingo2026.05 | 2.48 | 82.4 | 61.9 | 46.6 | 33.1 | 23.5 | |
| Diffusion PolicySampling=TapSampling2026.05 | 2.58 | 83.9 | 65.1 | 48.8 | 35.7 | 24.7 | |
| SuSIEArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=pixel-level2026.03 | 2.69 | 87 | 69 | 49 | 38 | 26 | |
| GR-12026.01 | 3.06 | 85.4 | 71.2 | 59.6 | 49.7 | 40.1 | |
| GR-12025.02 | 3.06 | 85.4 | 71.2 | 59.6 | 49.7 | 40.1 | |
| GR-1Architecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=latent-level2026.03 | 3.06 | 85.4 | 71.2 | 59.6 | 49.7 | 40.1 | |
| DiffPolicy2026.01 | 3.16 | 86.3 | 72.7 | 60.1 | 51.2 | 41.7 | |
| OpenVLAre-implementation=true2026.01 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLAreproduced=false2026.02 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLAArchitecture Paradigm (VLM vs VPM)=VLM, Representation Level (pixel-level vs latent-level)=N/A2026.03 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLASize=7B2026.04 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLA2026.05 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLA2026.05 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLA2026.06 | 3.27 | 91.3 | 77.8 | 62 | 52.1 | 43.5 | |
| OpenVLASampling=Original2026.05 | 3.3 | 93.4 | 78.2 | 64.1 | 52.2 | 42.4 | |
| 3D Diffusion Actor2025.02 | 3.35 | 93.8 | 80.3 | 66.2 | 53.3 | 41.2 | |
| VidManArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=latent-level2026.03 | 3.42 | 91.5 | 76.4 | 68.2 | 59.2 | 46.7 | |
| MDT-V w/ SmoothQuantBit-width=W4A4, Model Size=11.26 MB2026.04 | 3.43 | — | — | — | — | — | |
| MDT-V w/ AWQBit-width=W4A4, Model Size=11.26 MB2026.04 | 3.48 | — | — | — | — | — | |
| GR-1*Size=0.2B2026.04 | 3.51 | 91 | 79.2 | 68.5 | 60.3 | 52 | |
| OpenVLASampling=TapSampling2026.05 | 3.51 | 94.5 | 80.9 | 68.6 | 57.7 | 48.8 | |
| CLOVERArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=pixel-level2026.03 | 3.53 | 96 | 83.5 | 70.8 | 57.5 | 45.4 | |
| MDT-V w/ AWQBit-width=W8A8, Model Size=22.52 MB2026.04 | 3.55 | — | — | — | — | — | |
| MDT-V w/ SmoothQuantBit-width=W8A8, Model Size=22.52 MB2026.04 | 3.57 | — | — | — | — | — | |
| MDT-V w/ BatchQuantBit-width=W8A8, Model Size=22.52 MB2026.04 | 3.58 | — | — | — | — | — | |
| MDT-V w/ BatchQuantBit-width=W4A4, Model Size=11.26 MB2026.04 | 3.59 | — | — | — | — | — | |
| DitaSize=–2026.04 | 3.61 | 94.5 | 82.5 | 72.8 | 61.3 | 50 | |
| π02026.05 | 3.627 | 92.3 | 82.4 | 72.1 | 62.2 | 53.7 | |
| MDT-V w/ SQILBit-width=W4A4, Model Size=11.26 MB2026.04 | 3.63 | — | — | — | — | — | |
| RoboUniView2025.02 | 3.64 | 94.2 | 84.2 | 73.4 | 62.2 | 50.7 | |
| DC-QFABit-width=W8A8, Model Size=22.52 MB2026.04 | 3.64 | — | — | — | — | — | |
| AnchorRefine (GR-1)Size=0.22B2026.04 | 3.64 | 91.9 | 81.6 | 71.9 | 63.4 | 55.3 | |
| π0Architecture Paradigm (VLM vs VPM)=VLM, Representation Level (pixel-level vs latent-level)=N/A2026.03 | 3.65 | 93.7 | 83.2 | 74 | 62.9 | 51 | |
| MDT-V w/ BatchQuantBit-width=FP8, Model Size=22.52 MB2026.04 | 3.65 | — | — | — | — | — | |
| SkillDiffuser2026.01 | 3.66 | 94.4 | 82.7 | 72.1 | 62.4 | 55.4 | |
| RoboDualreproduced=false2026.02 | 3.66 | 94.4 | 82.7 | 72.1 | 62.4 | 54.4 | |
| DC-QFABit-width=W4A4, Model Size=11.26 MB2026.04 | 3.66 | — | — | — | — | — | |
| RoboDual2026.05 | 3.66 | 94.4 | 82.7 | 72.1 | 62.4 | 54.4 | |
| RoboDual2026.05 | 3.66 | 94.4 | 82.7 | 72.1 | 62.4 | 54.4 | |
| DC-QFABit-width=FP8, Model Size=22.52 MB2026.04 | 3.67 | — | — | — | — | — | |
| MDT-V w/ SQILBit-width=W8A8, Model Size=22.52 MB2026.04 | 3.67 | — | — | — | — | — | |
| MDT-VBit-width=FP16, Model Size=45.04 MB2026.04 | 3.72 | — | — | — | — | — | |
| UniVLA2026.01 | 3.8 | 95.5 | 85.8 | 75.4 | 66.9 | 56.5 | |
| UniVLAreproduced=false2026.02 | 3.8 | 95.5 | 85.8 | 75.4 | 66.9 | 56.5 | |
| UniVLASize=9B2026.04 | 3.8 | 95.5 | 85.8 | 75.4 | 66.9 | 56.5 | |
| UniVLA2026.05 | 3.8 | 95.5 | 85.8 | 75.4 | 66.9 | 56.5 | |
| UniVLA2026.05 | 3.8 | 95.4 | 85.5 | 75.4 | 66.9 | 56.5 | |
| UniVLA2026.06 | 3.8 | 95.5 | 85.8 | 75.4 | 66.9 | 56.5 | |
| π0.5Architecture Paradigm (VLM vs VPM)=VLM, Representation Level (pixel-level vs latent-level)=N/A2026.03 | 3.84 | 92.7 | 84.3 | 76.7 | 68.8 | 61.3 | |
| π0Size=3B2026.04 | 3.87 | 94.3 | 87 | 77.9 | 68.5 | 59.4 | |
| π0.52026.05 | 3.875 | 94.4 | 86 | 76.4 | 69.7 | 61 | |
| OpenVLA-OFT2026.05 | 3.917 | 94.2 | 86.4 | 78 | 70.4 | 62.7 | |
| MoDEre-implementation=true2026.01 | 3.92 | 95.8 | 88.1 | 79.3 | 70.8 | 62.4 | |
| π0reproduced=false2026.02 | 3.92 | 93.8 | 85 | 76.7 | 68.1 | 59.9 | |
| π02026.05 | 3.92 | 93.8 | 85 | 76.7 | 68.1 | 59.9 | |
| ReconVLAreproduced=false2026.02 | 3.95 | 95.6 | 87.6 | 76.9 | 69.3 | 64.1 | |
| ReconVLA2026.05 | 3.95 | 95.6 | 87.6 | 76.9 | 69.3 | 64.1 | |
| ReconVLA2026.05 | 3.95 | 95.6 | 87.6 | 76.9 | 69.3 | 64.1 | |
| SeeRversion=base2025.02 | 3.98 | 94.4 | 87.2 | 79.9 | 72.2 | 64.3 | |
| SeerArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=latent-level2026.03 | 3.98 | 94.4 | 87.2 | 79.9 | 72.2 | 64.3 | |
| SeerSize=0.3B2026.04 | 3.98 | 94.4 | 87.2 | 79.9 | 72.2 | 64.9 | |
| GR-MG2025.02 | 4.04 | 96.8 | 89.3 | 81.5 | 72.7 | 64.4 | |
| HARP-VLA (L2)Alignment loss type=L2, Freezing vision encoder=Yes2026.05 | 4.044 | 95.8 | 89.7 | 81.3 | 72.8 | 64.8 | |
| X-VLA + FASTERFASTER=true2026.03 | 4.058 | 97.7 | 91.1 | 81.2 | 72.1 | 63.7 | |
| UP-VLAreproduced=false2026.02 | 4.08 | 92.8 | 86.5 | 81.5 | 76.9 | 69.9 | |
| UP-VLASize=1.3B, Pretraining=true2026.03 | 4.08 | 92.8 | 86.5 | 81.5 | 76.9 | 69.9 | |
| UP-VLA2025.02 | 4.08 | 92.8 | 86.5 | 81.5 | 76.9 | 69.9 | |
| 2D Foresighttraining=scratch2025.02 | 4.08 | 95.6 | 90.5 | 83.1 | 74.8 | 64.2 | |
| UP-VLA2026.05 | 4.08 | 92.8 | 86.5 | 81.5 | 76.9 | 69.9 | |
| RoboFlamingo2026.01 | 4.09 | 96.4 | 89.6 | 82.4 | 74 | 66 | |
| Robo-FlamingoTask=ABCD→D2026.03 | 4.09 | 96.4 | 89.6 | 82.4 | 74 | 66 | |
| AtomicVLASize=–2026.04 | 4.09 | 95 | 87.8 | 81.9 | 75 | 69.1 | |
| OpenVLA-OFTArchitecture Paradigm (VLM vs VPM)=VLM, Representation Level (pixel-level vs latent-level)=N/A2026.03 | 4.1 | 96.3 | 89.1 | 82.4 | 75.8 | 66.5 | |
| DeerTask=ABCD→D2026.03 | 4.13 | 98.2 | 90.2 | 82.1 | 75.9 | 67 | |
| 3D Foresighttraining=scratch2025.02 | 4.15 | 96.2 | 91.1 | 84.4 | 77.1 | 68.7 | |
| X-VLA*FASTER=false, Cluster Evaluated=true2026.03 | 4.151 | 95.7 | 89.8 | 82.4 | 77 | 70.2 | |
| MoDEre-implementation=true2026.01 | 4.19 | 95.4 | 89.1 | 83.8 | 78.5 | 73.4 | |
| GR-12026.01 | 4.21 | 94.9 | 89.6 | 84.4 | 78.9 | 73.1 | |
| GR-1Task=ABCD→D2026.03 | 4.21 | 94.9 | 89.6 | 84.4 | 78.9 | 73.1 | |
| π0.5reproduced=true2026.05 | 4.21 | 94.8 | 88.9 | 84.1 | 79.7 | 72.1 | |
| 3D Foresighttraining=pre-trained2025.02 | 4.23 | 96.9 | 92 | 85.7 | 78.8 | 71.3 | |
| ReconVLATask=ABCD→D2026.03 | 4.23 | 98 | 90 | 84.5 | 78.5 | 70.5 | |
| RoboVLMreproduced=false2026.02 | 4.25 | 98 | 93.6 | 85.4 | 77.8 | 70.4 | |
| RoboVLMsSize=1.6B, Pretraining=true2026.03 | 4.25 | 98 | 93.6 | 85.4 | 77.8 | 70.4 | |
| RoboVLM2026.05 | 4.25 | 98 | 93.6 | 85.4 | 77.8 | 70.4 | |
| HARP-VLA (w/o F.)Alignment loss type=SRPD, Freezing vision encoder=No2026.05 | 4.25 | 98.8 | 93.9 | 86.1 | 77.7 | 68.5 | |
| π0.5reproduced=true2026.02 | 4.26 | 94.4 | 88.4 | 85.3 | 80.1 | 76.1 | |
| UniVLA*Task=ABCD→D, historical frames=false2026.03 | 4.26 | 94.8 | 90.6 | 86.2 | 83.4 | 69 | |
| Seerreproduced=false2026.02 | 4.28 | 96.3 | 91.6 | 86.1 | 80.3 | 74 | |
| SeerLargeArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=latent-level2026.03 | 4.28 | 96.3 | 91.6 | 86.1 | 80.3 | 74 | |
| VPPArchitecture Paradigm (VLM vs VPM)=VPM, Representation Level (pixel-level vs latent-level)=latent-level2026.03 | 4.28 | 96 | 91.3 | 86.4 | 80.4 | 74.7 | |
| Seer2026.05 | 4.28 | 96.3 | 91.6 | 86.1 | 80.3 | 74 |