Software Engineering on SWE-Bench Verified (Pass@K)
84Pass@1Mini-SWE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Mini-SWEModel Setup=DeepSeek-V3.22026.02 | 84 | 89 | 54.76 | |
| AORCHESTRAModel Setup=Gemini-3-Flash2026.02 | 82 | 86 | 71.62 | |
| AORCHESTRAModel Setup=DeepSeek-V3.22026.02 | 76 | 82 | 58.43 | |
| CCABackbone Model=Claude 4 Sonnet, Scaffold=CCA2025.12 | 74.6 | — | — | |
| OpenHandsBackbone Model=Claude 4 Sonnet, Scaffold=OpenHands2025.12 | 72.8 | — | — | |
| mini-SWE-AgentBackbone Model=Claude 4.5 Sonnet, Scaffold=mini-SWE-Agent2025.12 | 70.6 | — | — | |
| AORCHESTRAModel Setup=Claude-4.5-haiku2026.02 | 70 | 84 | 55.44 | |
| OpenHandsModel Setup=Claude-4.5-haiku2026.02 | 68 | 83 | 45.13 | |
| SWE-AgentBackbone Model=Claude 4 Sonnet, Scaffold=SWE-Agent2025.12 | 66.6 | — | — | |
| ReActModel Setup=Gemini-3-Flash2026.02 | 64 | 82 | 47.22 | |
| ReActModel Setup=Claude-4.5-haiku2026.02 | 63 | 87 | 43.62 | |
| OpenHandsModel Setup=DeepSeek-V3.22026.02 | 60 | 75 | 48.35 | |
| Mini-SWEModel Setup=Gemini-3-Flash2026.02 | 56 | 85 | 49.49 | |
| ReActModel Setup=DeepSeek-V3.22026.02 | 48 | 87 | 38.23 | |
| OpenHandsModel Setup=Gemini-3-Flash2026.02 | 48 | 66 | 48.49 | |
| Mini-SWEModel Setup=Claude-4.5-haiku2026.02 | 44 | 83 | 36.3 | |
| Claude CodeModel Setup=Claude-4.5-haiku2026.02 | 25 | 41 | 29.65 | |
| Claude CodeModel Setup=Gemini-3-Flash2026.02 | 22 | 42 | 27.43 | |
| AgentKB#Memories=5,8992026.04 | — | 72 | — | |
| MTL#Memories=4312026.04 | — | 77 | — | |
| ReasoningBank#Memories=972026.04 | — | 75 | — | |
| Zeroshot#Memories=-2026.04 | — | 73 | — |