Autonomous Software Engineering on SWE-Bench Pro (test)
32.6Resolution RateSonnet
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| SonnetSubagent Config.=Sonnet, Main Agent Model=Claude Opus 4.62026.05 | 32.6 | 756 | 19 | 774 | 0.6 | 6 | 100 | |
| SFT-4BSubagent Config.=SFT-4B, Main Agent Model=Claude Opus 4.62026.05 | 32.4 | 727 | 17 | 727 | 1.1 | 17 | 98.5 | |
| OpusSubagent Config.=Opus, Main Agent Model=Claude Opus 4.62026.05 | 32 | 729 | 19 | 748 | 0.5 | 4 | 100 | |
| Terminus-4BSubagent Config.=Terminus-4B, Main Agent Model=Claude Opus 4.62026.05 | 31.5 | 730 | 18 | 730 | 1 | 14 | 97.9 | |
| Vanilla-4BSubagent Config.=Vanilla-4B, Main Agent Model=Claude Opus 4.62026.05 | 30.4 | 840 | 6 | 840 | 1.9 | 27 | 99 | |
| No SubagentSubagent Config.=No Subagent, Main Agent Model=Claude Opus 4.62026.05 | 30 | 836 | — | 836 | 3.8 | — | — |