Software Engineering Problem Solving on SWE-bench Verified (Pass@K & Cost)
78Average Success Rate @1DELM
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DELMBase Model=Claude Opus 4.62026.06 | 78 | 80.7 | 82.5 | 0.63 | |
| mini-SWE-agentBase Model=Claude Opus 4.62026.06 | 76.9 | 79.8 | 81.7 | 0.61 | |
| Claude CodeBase Model=Claude Opus 4.62026.06 | 76.1 | 79.5 | 81.3 | 0.69 | |
| AOrchestra-ParallelBase Model=Claude Opus 4.62026.06 | 75.2 | 78.1 | 79.7 | 0.73 | |
| AOrchestraBase Model=Claude Opus 4.62026.06 | 74.7 | 78.3 | 80.1 | 0.7 | |
| DELMBase Model=Gemini 3 Flash2026.06 | 65.7 | 72.9 | 77.4 | 0.12 | |
| AOrchestra-ParallelBase Model=Gemini 3 Flash2026.06 | 56.4 | 63.2 | 71.8 | 0.25 | |
| AOrchestraBase Model=Gemini 3 Flash2026.06 | 55.2 | 64.5 | 73.2 | 0.24 | |
| mini-SWE-agentBase Model=Gemini 3 Flash2026.06 | 54.7 | 65.6 | 75.1 | 0.26 | |
| Claude CodeBase Model=Gemini 3 Flash2026.06 | 49.3 | 57.1 | 66.3 | — |