Error Localization on SWE-bench
70Exact Match (EM)REFLECT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| REFLECTAuditor model=gpt-5.2, Verification source=Oracle2026.06 | 70 | 93.3 | |
| REFLECT (proxy)Auditor model=gpt-5.2, Verification source=Proxy2026.06 | 66.1 | 80 | |
| Reflexion+GTAuditor model=gpt-5.2, Ground Truth availability=Yes2026.06 | 45.2 | 87.1 | |
| Reflexion–noGTAuditor model=gpt-5.2, Ground Truth availability=No2026.06 | 41.9 | 83.9 | |
| AgentRxAuditor model=gpt-5.22026.06 | 41.9 | 83.9 | |
| ThinkPRMAuditor model=gpt-5.22026.06 | 38.7 | 80.6 | |
| AAOAuditor model=gpt-5.22026.06 | 33.3 | 83.3 | |
| BSAuditor model=gpt-5.22026.06 | 33.3 | 76.7 | |
| SBSAuditor model=gpt-5.22026.06 | 26.7 | 80 | |
| AAO (Opus 4.6)Auditor model=Claude Opus 4.62026.06 | 19.4 | 77.4 | |
| ICSAuditor model=gpt-5.22026.06 | 18.2 | 90.9 | |
| DoVer+GTAuditor model=gpt-5.2, Ground Truth availability=Yes2026.06 | 12.9 | 80.6 |