Vulnerability Reproduction on CyberGym (evaluation)
1,600Total RolloutsGPT-5.4 mini Best-of-8
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.4 mini Best-of-8Scaffold=Codex Scaffold, Backbone=GPT-5.4 mini, Protocol=Best-of-82026.07 | 1,600 | 7 | 93 | 2 | 13 | 85 | 42.5 | 35.9 | |
| GPT-5.4 mini Best-of-8 + PAGENTScaffold=Codex Scaffold, Backbone=GPT-5.4 mini, Protocol=Best-of-8 + PAGENT2026.07 | 1,600 | 9 | 100 | 0 | 5 | 86 | 43 | 36.3 | |
| GPT-5.5 Best-of-8Scaffold=Codex Scaffold, Backbone=GPT-5.5, Protocol=Best-of-82026.07 | 1,600 | 4 | 49 | 1 | 20 | 126 | 63 | 56.1 | |
| GPT-5.5 Best-of-8 + PAGENTScaffold=Codex Scaffold, Backbone=GPT-5.5, Protocol=Best-of-8 + PAGENT2026.07 | 1,600 | 6 | 35 | 1 | 17 | 141 | 70.5 | 63.8 | |
| GLM 5.1 Best-of-8Scaffold=Claude Code Scaffold, Backbone=GLM 5.1, Protocol=Best-of-82026.07 | 1,600 | 6 | 77 | 1 | 7 | 109 | 54.5 | 47.6 | |
| GLM 5.1 Best-of-8 + PAGENTScaffold=Claude Code Scaffold, Backbone=GLM 5.1, Protocol=Best-of-8 + PAGENT2026.07 | 1,600 | 2 | 62 | 2 | 12 | 122 | 61 | 54.1 | |
| GPT-5.4 mini Iterative ImprovementScaffold=Codex Scaffold, Backbone=GPT-5.4 mini, Protocol=Iterative Improvement2026.07 | 1,026 | 0 | 86 | 0 | 8 | 106 | 53 | 46.1 | |
| GPT-5.4 mini + Base PlannerScaffold=Codex Scaffold, Backbone=GPT-5.4 mini, Protocol=Base Planner2026.07 | 924 | 0 | 99 | 0 | 11 | 90 | 45 | 38.3 | |
| GPT-5.4 mini Executor + MastermindScaffold=Codex Scaffold, Backbone=GPT-5.4 mini, Protocol=Mastermind (frozen strategy planner)2026.07 | 891 | 2 | 26 | 40 | 12 | 120 | 60 | 53.1 | |
| GLM 5.1 Iterative ImprovementScaffold=Claude Code Scaffold, Backbone=GLM 5.1, Protocol=Iterative Improvement2026.07 | 842 | 0 | 56 | 2 | 8 | 134 | 67 | 60.2 | |
| GPT-5.5 Iterative ImprovementScaffold=Codex Scaffold, Backbone=GPT-5.5, Protocol=Iterative Improvement2026.07 | 753 | 0 | 28 | 0 | 18 | 154 | 77 | 70.7 | |
| GLM 5.1 Executor + Base PlannerScaffold=Claude Code Scaffold, Backbone=GLM 5.1, Protocol=Base Planner2026.07 | 750 | 0 | 51 | 1 | 31 | 117 | 58.5 | 51.6 | |
| GLM 5.1 Executor + MastermindScaffold=Claude Code Scaffold, Backbone=GLM 5.1, Protocol=Mastermind (frozen strategy planner)2026.07 | 719 | 0 | 21 | 19 | 18 | 142 | 71 | 64.4 | |
| GPT-5.5 Executor + Base PlannerScaffold=Codex Scaffold, Backbone=GPT-5.5, Protocol=Base Planner2026.07 | 619 | 1 | 34 | 0 | 20 | 145 | 72.5 | 65.9 | |
| GPT-5.5 Executor + MastermindScaffold=Codex Scaffold, Backbone=GPT-5.5, Protocol=Mastermind (frozen strategy planner)2026.07 | 560 | 1 | 10 | 15 | 5 | 169 | 84.5 | 78.8 |