Failure Detection on RoboFail
67AccuracyGemini 2.0 Flash
Evaluation results
| Method | Links | |
|---|---|---|
| Gemini 2.0 FlashModel Source=Closed-Source2026.03 | 67 | |
| PRIMO R1Model Source=Ours2026.03 | 67 | |
| Cosmos-Reason1-56BModel Source=Open-Source2026.03 | 66.2 | |
| Nemotron-H-56BModel Source=Open-Source2026.03 | 64 | |
| GPT-4oModel Source=Closed-Source2026.03 | 63 | |
| PRIMO (RL)Model Source=Ours, Training Stage=RL2026.03 | 63 | |
| OpenAI o1Model Source=Closed-Source2026.03 | 61 | |
| Cosmos-Reason1-7BModel Source=Open-Source2026.03 | 60 | |
| Claude-haiku-4.5Model Source=Closed-Source2026.03 | 59 | |
| Qwen2.5-VL-7BModel Source=Open-Source2026.03 | 57.6 | |
| PRIMO (SFT)Model Source=Ours, Training Stage=SFT2026.03 | 51 |