Failure Detection on DSMF-CALVIN (test)
90.64AccuracyI-FailSense
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| I-FailSenseProtocol=Stages 1 + 2 (ours), #PoV=12025.09 | 90.64 | 88.5 | 94.34 | 91.32 | |
| I-FailSenseProtocol=Stages 1 + 2 (ours), #PoV=22025.09 | 88.18 | 85.96 | 92.45 | 89.09 | |
| I-FailSenseProtocol=LoRA only, #PoV=12025.09 | 85.71 | 86.67 | 85.85 | 86.26 | |
| I-FailSenseProtocol=LoRA only, #PoV=22025.09 | 82.27 | 83.65 | 82.08 | 82.86 | |
| Qwen2.5-VL 7BProtocol=zero-shot, #PoV=22025.09 | 69.46 | 65.07 | 89.62 | 75.4 | |
| Qwen2.5-VL 7BProtocol=zero-shot, #PoV=12025.09 | 67.98 | 63.23 | 92.45 | 75.1 | |
| GPT-4oProtocol=zero-shot, #PoV=22025.09 | 63.05 | 77.19 | 41.51 | 53.99 | |
| GPT-4oProtocol=zero-shot, #PoV=12025.09 | 57.64 | 81.25 | 24.53 | 37.68 | |
| PaliGemma2-mix 3BProtocol=zero-shot, #PoV=22025.09 | 52.71 | 53.29 | 76.42 | 62.79 | |
| PaliGemma2-mix 3BProtocol=zero-shot, #PoV=12025.09 | 47.29 | 49.61 | 59.43 | 54.08 |