Robotic Failure Analysis on RoboFAC 1.0 (mixed simulated and real-world)
82.74Task Success Rate (Short Horizon)RoboFAC-7B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| RoboFAC-7BParameters=7B2025.05 | 82.74 | 84.92 | 81.78 | 83.28 | 68.94 | 79.1 | |
| RoboFAC-3BParameters=3B2025.05 | 81.66 | 84.67 | 79.32 | 83.02 | 63.29 | 76.8 | |
| Gemini-2.02025.05 | 63.32 | 53.23 | 45.67 | 48.91 | 41.72 | 51.11 | |
| GPT-4o2025.05 | 61.5 | 53.81 | 42.46 | 45.82 | 65.89 | 57.42 | |
| Qwen-2.5-VL-3BParameters=3B2025.05 | 40.99 | 27.82 | 25.18 | 28.94 | 17.36 | 27.82 | |
| Qwen-2.5-VL-7BParameters=7B2025.05 | 14.26 | 11.73 | 38.84 | 18 | 50.96 | 27.47 |