# Agent Evaluation (Robustness and Success Rate) on Terminal Bench 77 tasks (held-out)

Source: [Agent Evaluation (Robustness and Success Rate) on Terminal Bench 77 tasks (held-out)](<https://www.sota2.com/research/sota/agent-evaluation-robustness-and-success-rate-on-terminal-bench-77-tasks-held-out>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-09T10:04:07.556Z

Evaluation of agent performance, robustness, and success rate across 77 held-out tasks within the Terminal Bench framework, measured by metrics such as Hinted/Unhinted ASR and Benign Pass rate.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Agent Evaluation (Robustness and Success Rate) |
| Dataset (including any stated split) | Terminal Bench 77 tasks (held-out) |
| Metric | Hinted ASR |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 4 of 4 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Hinted ASR](<https://www.sota2.com/research/sota/agent-evaluation-robustness-and-success-rate-on-terminal-bench-77-tasks-held-out.md?metric=Hinted%20ASR>): Higher is better; unit: PERCENT.

- [Unhinted ASR](<https://www.sota2.com/research/sota/agent-evaluation-robustness-and-success-rate-on-terminal-bench-77-tasks-held-out.md?metric=Unhinted%20ASR>): Higher is better; unit: PERCENT.

- [Benign Pass Rate](<https://www.sota2.com/research/sota/agent-evaluation-robustness-and-success-rate-on-terminal-bench-77-tasks-held-out.md?metric=Benign%20Pass%20Rate>): Higher is better; unit: PERCENT.

- [Delta Percentage Points (∆ pp)](<https://www.sota2.com/research/sota/agent-evaluation-robustness-and-success-rate-on-terminal-bench-77-tasks-held-out.md?metric=Delta%20Percentage%20Points%20(%E2%88%86%20pp)>): Higher is better; unit: SCALAR.

- [P-value](<https://www.sota2.com/research/sota/agent-evaluation-robustness-and-success-rate-on-terminal-bench-77-tasks-held-out.md?metric=P-value>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Hinted ASR | Result date | Report paper |
| --- | --- | --- | --- |
| Original Verifier | 100 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Hardened Verifier | 92.2 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Original Verifier | 50.4 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Hardened Verifier | 39.4 | Not recorded | 6a279e7b90966ade919bfbc2 |

### Original Verifier

Hinted ASR: 100. Result ID: 6a27e43efbc373592c252f93.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

Recorded evaluation configuration: {"Evaluation Level":"Task-level","Verifier State":"Original","Hint attempts per task":"8","Unhinted attempts per task":"3","Solver attempts per task":"3"}. This record does not assert independent reproduction.

### Hardened Verifier

Hinted ASR: 92.2. Result ID: 6a27e43efbc373592c252f94.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Evaluation Level":"Task-level","Verifier State":"Hardened","Methodology":"hacker-fixer loop","Hint attempts per task":"8","Unhinted attempts per task":"3","Solver attempts per task":"3"}. This record does not assert independent reproduction.

### Original Verifier

Hinted ASR: 50.4. Result ID: 6a27e43efbc373592c252f91.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

Recorded evaluation configuration: {"Evaluation Level":"Trajectory-level","Verifier State":"Original"}. This record does not assert independent reproduction.

### Hardened Verifier

Hinted ASR: 39.4. Result ID: 6a27e43efbc373592c252f92.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Evaluation Level":"Trajectory-level","Verifier State":"Hardened","Methodology":"hacker-fixer loop"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.