Robotic Manipulation on VIMA-Bench 1.0 (test)
81.5L1 ScoreVIMA
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| VIMAInput Representation=Object tokens (cropped images and bounding boxes)2023.12 | 81.5 | 81.5 | 78.7 | 48.6 | 72.6 | — | |
| iSTARVariant=Full model2026.02 | 78.6 | 79.2 | 78.3 | 44.3 | — | — | |
| iSTARVariant=w/o Concept Alignment2026.02 | 77.8 | 77.5 | 77 | 40 | — | — | |
| iSTARVariant=w/o Order Entropy2026.02 | 76.4 | 77.6 | 76.3 | 42.7 | — | — | |
| iSTARVariant=w/o Graph2026.02 | 74.8 | 76 | 74.9 | 37.7 | — | — | |
| VIMAModel size=Large2026.02 | 73.9 | 74.5 | 72 | 59 | — | — | |
| VIMAModel size=Small2026.02 | 72.4 | 73.8 | 71.1 | 35 | — | — | |
| iSTARVariant=Stacked VIMA2026.02 | 72.4 | 73.6 | 72.4 | 34 | — | — | |
| UNIFIED-IO 2 XXLModel Scale=XXL2023.12 | 71.3 | 70.4 | 68 | 15.5 | 56.3 | — | |
| UNIFIED-IO 2 XLModel Scale=XL2023.12 | 70.3 | 69.8 | 64.5 | 13.1 | 54.2 | — | |
| UNIFIED-IO 2 LModel Scale=L2023.12 | 66.9 | 63.8 | 57.5 | 12.6 | 50.2 | — | |
| VIMA-GatoInput Representation=Image patch tokens (ViT)2023.12 | 57 | 53.9 | 45.6 | 13.5 | 42.5 | — | |
| VIMA-FlamingoInput Representation=Image patch tokens (ViT) + downsampled by perceiver module2023.12 | 47.4 | 46 | 40.7 | 12.1 | 36.6 | — | |
| VIMA-GPTInput Representation=Single image token (ViT)2023.12 | 46.9 | 46.9 | 42.2 | 12.1 | 37 | — | |
| LLaRA-dist#Data=1K2025.03 | — | — | — | — | — | 53.9 | |
| LLaRA-dist#Data=10K2025.03 | — | — | — | — | — | 83.4 | |
| LLaRA-dist#Data=100K2025.03 | — | — | — | — | — | 89.5 | |
| LLaRA-sft#Data=1K2025.03 | — | — | — | — | — | 49.6 | |
| LLaRA-sft#Data=10K2025.03 | — | — | — | — | — | 82.3 | |
| LLaRA-sft#Data=100K2025.03 | — | — | — | — | — | 88.5 | |
| LLaRA-vocab#Data=1K2025.03 | — | — | — | — | — | 50.8 | |
| LLaRA-vocab#Data=10K2025.03 | — | — | — | — | — | 81 | |
| LLaRA-vocab#Data=100K2025.03 | — | — | — | — | — | 87 | |
| RT-2#Data=1K2025.03 | — | — | — | — | — | 1.9 | |
| RT-2#Data=10K2025.03 | — | — | — | — | — | 21.9 | |
| RT-2#Data=100K2025.03 | — | — | — | — | — | 73.1 |