Issue Resolution on SWE-bench Verified (test)
77.2Pass RateClaude-4.5-Sonnet
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Claude-4.5-SonnetAgent Framework=ReAct Agent, Model Size=100B+, Scaffold=OpenHands2025.12 | 77.2 | — | — | — | — | |
| GPT-5.1Agent Framework=ReAct Agent, Model Size=100B+, Scaffold=OpenHands2025.12 | 76.3 | — | — | — | — | |
| Gemini-3-ProAgent Framework=ReAct Agent, Model Size=100B+, Scaffold=OpenHands2025.12 | 76.2 | — | — | — | — | |
| Qwen3-Coder-480B-A35BAgent Framework=ReAct Agent, Model Size=35/480B, Scaffold=OpenHands2025.12 | 69.6 | — | — | — | — | |
| Kimi-K2 (Sub-Agent)Localization Mode=Sub-Agent, Localization Model=FuseSearch-4B2026.01 | 68.7 | 31.9 | 290 | 713,000 | — | |
| Kimi-K2 (No Localization)Localization Mode=None2026.01 | 68.4 | 41.1 | 312 | 1,053,000 | — | |
| Kimi-K2 (Pre-Search)Localization Mode=Pre-Search, Localization Model=FuseSearch-4B2026.01 | 68.1 | 31.6 | 223 | 562,000 | — | |
| DeepSeek-V3.1Agent Framework=ReAct Agent, Model Size=37/671B, Scaffold=OpenHands2025.12 | 61 | — | — | — | — | |
| Gemini-2.5-ProAgent Framework=ReAct Agent, Model Size=100B+, Scaffold=OpenHands2025.12 | 59.6 | — | — | — | — | |
| GLM-4.5-AirAgent Framework=ReAct Agent, Model Size=12/106B, Scaffold=OpenHands2025.12 | 57.6 | — | — | — | — | |
| SWE-CompressorAgent Framework=Folding Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 57.6 | — | — | — | — | |
| Seed-OSS-36BAgent Framework=ReAct Agent, Model Size=36B, Scaffold=OpenHands2025.12 | 55.2 | — | — | — | — | |
| FrogBoss-32BAgent Framework=ReAct Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 54.6 | — | — | — | — | |
| Threshold-Compression AgentAgent Framework=Summary Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 53.8 | — | — | — | — | |
| Claude-3.5-SonnetAgent Framework=ReAct Agent, Model Size=100B+, Scaffold=OpenHands2025.12 | 53 | — | — | — | — | |
| SWE-Mirror-LM-32BAgent Framework=ReAct Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 52.2 | — | — | — | — | |
| ReAct AgentAgent Framework=Summary Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 49.8 | — | — | — | — | |
| DeepSeek-R1-0528Agent Framework=ReAct Agent, Model Size=37/671B, Scaffold=OpenHands2025.12 | 45.6 | — | — | — | — | |
| DeepSWE-32B-PreviewAgent Framework=ReAct Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 42.2 | — | — | — | — | |
| Llama3-SWE-RL-70BAgent Framework=ReAct Agent, Model Size=70B, Scaffold=OpenHands2025.12 | 41 | — | — | — | — | |
| SWE-agent-LM-32BAgent Framework=ReAct Agent, Model Size=32B, Scaffold=SWE-agent2025.12 | 40.2 | — | — | — | — | |
| GPT-4oAgent Framework=ReAct Agent, Model Size=100B+, Scaffold=Agentless2025.12 | 38.8 | — | — | — | — | |
| R2E-Gym-32BAgent Framework=ReAct Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 34.4 | — | — | — | — | |
| Qwen3-235B-A22BAgent Framework=ReAct Agent, Model Size=22/235B, Scaffold=OpenHands2025.12 | 34.4 | — | — | — | — | |
| SWE-Fixer-72BAgent Framework=ReAct Agent, Model Size=72B, Scaffold=SWE-Fixer2025.12 | 32.8 | — | — | — | — | |
| Lingma-SWE-GPT-72BAgent Framework=ReAct Agent, Model Size=72B, Scaffold=SWE-SynInfer2025.12 | 28.8 | — | — | — | — | |
| CWM (Meta)Size=32B, SWE Training Data=3M trajs from 3.15K repos2026.01 | 22.22 | — | — | — | 15.88 | |
| GPT-4.1-miniSize=[10, 100]B, SWE Training Data=-2026.01 | 21.79 | — | — | — | 17.85 | |
| SWE-Gym-32BAgent Framework=ReAct Agent, Model Size=32B, Scaffold=OpenHands2025.12 | 20.6 | — | — | — | — | |
| SWE-SPOTSize=4B, SWE Training Data=8K trajs in one target repo2026.01 | 19.34 | — | — | — | 17.12 | |
| Mini-Coder-4BSize=4B, SWE Training Data=400K trajs from 128 repos2026.01 | 18.76 | — | — | — | 8.7 | |
| Qwen3-Coder-30BSize=32B, SWE Training Data=-2026.01 | 16.74 | — | — | — | 11.56 | |
| Gemini-2.5-Flash-LiteSize=[0.1, 10]B, SWE Training Data=-2026.01 | 8.8 | — | — | — | 7.46 | |
| GPT-5-nanoSize=[0.1, 10]B, SWE Training Data=-2026.01 | 6.49 | — | — | — | 2.97 | |
| Qwen3-4B-Instruct-2507Size=4B, SWE Training Data=-2026.01 | 3.32 | — | — | — | 3.2 | |
| Gemma-3-27b-itSize=27B, SWE Training Data=-2026.01 | 1.29 | — | — | — | 1.34 |