Procedural mistake detection on Assembly101-O
59.6F1 ScoreESTANet
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ESTANet2026.06 | 59.6 | — | — | 49.6 | 69.5 | |
| DTGL2026.06 | 53.5 | — | — | 28.1 | 78.9 | |
| PREGOStep Recog.=Oracle, Step Antic.=LLAMA2024.04 | 46.3 | 30.7 | 94 | — | — | |
| MSG22026.06 | 46.2 | — | — | 33.2 | 59.1 | |
| PREGOStep Recog.=Oracle, Step Antic.=GPT-3.52024.04 | 42.1 | 29.2 | 75.8 | — | — | |
| PREGOStep Recog.=MiniRoad [2], Step Antic.=LLAMA2024.04 | 41.8 | 27.8 | 84.1 | — | — | |
| PREGOStep Recog.=OadTR [35], Step Antic.=LLAMA2024.04 | 35.8 | 22.1 | 94.2 | — | — | |
| MSGI2026.06 | 33.1 | — | — | 43.5 | 22.7 | |
| PREGO2026.06 | 32.5 | — | — | 41.8 | 23.1 | |
| BERTStep Recog.=Oracle2024.04 | 31.8 | 78.2 | 20 | — | — | |
| PREGOStep Recog.=MiniRoad [2], Step Antic.=GPT-3.52024.04 | 27.3 | 16.2 | 87.5 | — | — | |
| One-step memoryStep Recog.=Oracle2024.04 | 21.3 | 16.3 | 30.7 | — | — | |
| OadTR for MDStep Recog.=OadTR [35], Step Antic.=OadTR [35]2024.04 | 20.7 | 24.3 | 18.1 | — | — |