Verifier Robustness Evaluation on KernelBench task 001 (held-out)
62Hinted ASRNo defense
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| No defenseAdversary Model=Gemini 3.1 Pro2026.06 | 62 | 76 | 98 | |
| Blind, no poolAdversary Model=Gemini 3.1 Pro, Robust at=Iter. 22026.06 | 52 | 20 | 98 | |
| No defenseAdversary Model=Claude Opus 4.72026.06 | 44 | 61 | 90 | |
| Blind, no poolAdversary Model=Claude Opus 4.7, Robust at=Iter. 22026.06 | 44 | 61 | 90 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 5, Autopatch=true2026.06 | 41 | 61 | 98 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 2, Autopatch=true2026.06 | 17 | 2 | 98 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 5, Autopatch=true2026.06 | 15 | 2 | 92 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 5, Autopatch=false2026.06 | 15 | 2 | 0 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 2, Autopatch=true2026.06 | 11 | 2 | 90 | |
| Blind + poolAdversary Model=Gemini 3.1 Pro, Robust at=Iter. 52026.06 | 6 | 0 | 98 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 5, Autopatch=false2026.06 | 5 | 0 | 0 | |
| Blind + poolAdversary Model=Claude Opus 4.7, Robust at=Iter. 52026.06 | 2 | 6 | 90 | |
| Aware, no poolAdversary Model=Gemini 3.1 Pro, Robust at=Iter. 82026.06 | 2 | 0 | 98 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 8, Autopatch=true2026.06 | 0 | 0 | 98 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 8, Autopatch=true2026.06 | 0 | 0 | 92 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 11, Autopatch=true2026.06 | 0 | 0 | 98 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 11, Autopatch=true2026.06 | 0 | 0 | 92 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 13 (final), Autopatch=true2026.06 | 0 | 0 | 98 | |
| Verifier-aware + pool (with autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 13 (final), Autopatch=true2026.06 | 0 | 0 | 92 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 8, Autopatch=false2026.06 | 0 | 0 | 0 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 8, Autopatch=false2026.06 | 0 | 0 | 0 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 11, Autopatch=false2026.06 | 0 | 0 | 94 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 11, Autopatch=false2026.06 | 0 | 0 | 22 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Gemini 3.1 Pro, Iteration=Iter. 13 (final), Autopatch=false2026.06 | 0 | 0 | 0 | |
| Verifier-aware + pool (no autopatch)Adversary Model=Claude Opus 4.7, Iteration=Iter. 13 (final), Autopatch=false2026.06 | 0 | 0 | 0 | |
| Aware, no poolAdversary Model=Claude Opus 4.7, Robust at=Iter. 82026.06 | 0 | 0 | 90 |