Agent Evaluation (Robustness and Success Rate) on Terminal Bench 77 tasks (held-out)
100Hinted ASROriginal Verifier
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Original VerifierEvaluation Level=Task-level, Verifier State=Original, Hint attempts per task=8, Unhinted attempts per task=3, Solver attempts per task=32026.06 | 100 | 55.8 | 84.4 | — | — | |
| Hardened VerifierEvaluation Level=Task-level, Verifier State=Hardened, Methodology=hacker-fixer loop, Hint attempts per task=8, Unhinted attempts per task=3, Solver attempts per task=32026.06 | 92.2 | 30.3 | 74 | — | — | |
| Original VerifierEvaluation Level=Trajectory-level, Verifier State=Original2026.06 | 50.4 | 39.2 | 76.1 | — | — | |
| Hardened VerifierEvaluation Level=Trajectory-level, Verifier State=Hardened, Methodology=hacker-fixer loop2026.06 | 39.4 | 16.7 | 65.2 | — | — |