# Verifier Robustness Evaluation on KernelBench task 001 (held-out)

Source: [Verifier Robustness Evaluation on KernelBench task 001 (held-out)](<https://www.sota2.com/research/sota/verifier-robustness-evaluation-on-kernelbench-task-001-held-out>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-09T10:04:07.451Z

Evaluation of verifier robustness performance on the held-out split of KernelBench task 001, measured by Hinted ASR, Unhinted ASR, and Benign pass rate.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Verifier Robustness Evaluation |
| Dataset (including any stated split) | KernelBench task 001 (held-out) |
| Metric | Hinted ASR |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 26 of 26 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Hinted ASR](<https://www.sota2.com/research/sota/verifier-robustness-evaluation-on-kernelbench-task-001-held-out.md?metric=Hinted%20ASR>): Higher is better; unit: PERCENT.

- [Unhinted ASR](<https://www.sota2.com/research/sota/verifier-robustness-evaluation-on-kernelbench-task-001-held-out.md?metric=Unhinted%20ASR>): Higher is better; unit: PERCENT.

- [Benign Pass Rate](<https://www.sota2.com/research/sota/verifier-robustness-evaluation-on-kernelbench-task-001-held-out.md?metric=Benign%20Pass%20Rate>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Hinted ASR | Result date | Report paper |
| --- | --- | --- | --- |
| No defense | 62 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Blind, no pool | 52 | Not recorded | 6a279e7b90966ade919bfbc2 |
| No defense | 44 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Blind, no pool | 44 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 41 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 17 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 15 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 15 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 11 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Blind + pool | 6 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 5 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Blind + pool | 2 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Aware, no pool | 2 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (with autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Verifier-aware + pool (no autopatch) | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |
| Aware, no pool | 0 | Not recorded | 6a279e7b90966ade919bfbc2 |

### No defense

Hinted ASR: 62. Result ID: 6a27e43efbc373592c252f71.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro"}. This record does not assert independent reproduction.

### Blind, no pool

Hinted ASR: 52. Result ID: 6a27e43efbc373592c252f85.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Robust at":"Iter. 2"}. This record does not assert independent reproduction.

### No defense

Hinted ASR: 44. Result ID: 6a27e43efbc373592c252f72.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7"}. This record does not assert independent reproduction.

### Blind, no pool

Hinted ASR: 44. Result ID: 6a27e43efbc373592c252f86.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Robust at":"Iter. 2"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 41. Result ID: 6a27e43efbc373592c252f75.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 5","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 17. Result ID: 6a27e43efbc373592c252f73.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 2","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 15. Result ID: 6a27e43efbc373592c252f76.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 5","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 15. Result ID: 6a27e43efbc373592c252f7e.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 5","Autopatch":"false"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 11. Result ID: 6a27e43efbc373592c252f74.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 2","Autopatch":"true"}. This record does not assert independent reproduction.

### Blind + pool

Hinted ASR: 6. Result ID: 6a27e43efbc373592c252f87.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Robust at":"Iter. 5"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 5. Result ID: 6a27e43efbc373592c252f7d.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 5","Autopatch":"false"}. This record does not assert independent reproduction.

### Blind + pool

Hinted ASR: 2. Result ID: 6a27e43efbc373592c252f88.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Robust at":"Iter. 5"}. This record does not assert independent reproduction.

### Aware, no pool

Hinted ASR: 2. Result ID: 6a27e43efbc373592c252f89.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Robust at":"Iter. 8"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f77.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 8","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f78.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 8","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f79.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 11","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f7a.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 11","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f7b.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 13 (final)","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (with autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f7c.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 13 (final)","Autopatch":"true"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f7f.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 8","Autopatch":"false"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f80.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 8","Autopatch":"false"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f81.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 11","Autopatch":"false"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f82.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 11","Autopatch":"false"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f83.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Gemini 3.1 Pro","Iteration":"Iter. 13 (final)","Autopatch":"false"}. This record does not assert independent reproduction.

### Verifier-aware + pool (no autopatch)

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f84.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Iteration":"Iter. 13 (final)","Autopatch":"false"}. This record does not assert independent reproduction.

### Aware, no pool

Hinted ASR: 0. Result ID: 6a27e43efbc373592c252f8a.

[Reporting paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a279e7b90966ade919bfbc2.md>)

[Implementation](<https://github.com/few-sh/harden-v0>)

Recorded evaluation configuration: {"Adversary Model":"Claude Opus 4.7","Robust at":"Iter. 8"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.