Software Engineering on SWE-bench Verified (Success Rate (%))
80.9Success RateClaude-Opus-4.5
Evaluation Results
| Method | Links | |
|---|---|---|
| Claude-Opus-4.52026.06 | 80.9 | |
| GPT-5.12026.06 | 76.3 | |
| Gemini-3-Pro2026.06 | 76.2 | |
| GLM-4.72026.06 | 73.8 | |
| DeepSeek-V3.22026.06 | 73.1 | |
| OpenHandsBase Model=GPT-5, Scaffold=OpenHands, Data=-2026.04 | 71.8 | |
| OpenHandsBase Model=Claude 4 Sonnet, Scaffold=OpenHands, Data=-2026.04 | 70.4 | |
| Qwen3-CoderBase Model=Qwen3-Coder-480B, Scaffold=OpenHands, Data=-2026.04 | 69.6 | |
| Kimi-K2-Instruct-09052026.06 | 69.2 | |
| Qwen3-Coder-480B-A35B-Instruct2026.06 | 67 | |
| SWE-agentBase Model=Claude 4 Sonnet, Scaffold=SWE-agent, Data=-2026.04 | 66.6 | |
| Kimi-K2Base Model=Kimi-K2-1T, Scaffold=OpenHands, Data=-2026.04 | 65.4 | |
| LoopCoder-v2R (number of refinement loops)=22026.06 | 64.4 | |
| GLM-4.5Base Model=GLM-4.5-355B, Scaffold=OpenHands, Data=-2026.04 | 64.2 | |
| Kimi-Dev-72B2026.06 | 60.4 | |
| CodeFuse-CGMBase Model=Qwen2.5-72B, Scaffold=Graph RAG, Data=200k issue-patch pairs2026.04 | 50.4 | |
| Best DGM agentAgent Stage=Best, Evaluation Protocol=Endpoint2026.06 | 50 | |
| ENTROPO-KTOBase Model=Qwen3-Coder-30B, Scaffold=R2EGym, Data=-2026.04 | 49.3 | |
| Kimi-DevBase Model=Qwen 2.5-72B, Scaffold=SWE-Agent, Data=150B tokens +2026.04 | 48.6 | |
| Qwen3-235B-A22B-Instruct-25072026.06 | 45.2 | |
| DeepSWE+AEMBase Model=Qwen3-32B2026.05 | 43.7 | |
| LoopCoder-v2R (number of refinement loops)=12026.06 | 43 | |
| DeepSWEBase Model=Qwen3-32B2026.05 | 42.3 | |
| DeepSWEBase Model=Qwen3-32B, Scaffold=R2EGym, Data=4.5k SWE tasks2026.04 | 42.2 | |
| SWE-RLBase Model=Llama-3.3-70B, Scaffold=Agentless Mini, Data=-2026.04 | 41 | |
| SWE-smithBase Model=Qwen2.5-Coder-32B, Scaffold=SWE-agent, Data=5k trajectories2026.04 | 40.2 | |
| Agentless-1.5Base Model=GPT-4o, Scaffold=Agentless, Data=-2026.04 | 38.8 | |
| Skywork-SWEBase Model=Qwen2.5-Coder, Scaffold=OpenHands, Data=8k trajectories2026.04 | 38 | |
| SWE-DevBase Model=Qwen2.5-Coder-32B, Scaffold=OpenHands, Data=19.3k trajectories2026.04 | 36.6 | |
| R2EGymBase Model=Qwen2.5-Coder-32B, Scaffold=R2EGym, Data=3.3k trajectories2026.04 | 34.4 | |
| SWE-fixerBase Model=Qwen2.5-72B, Scaffold=Pipline, Data=-2026.04 | 32.8 | |
| SWE-AGILE (SFT)Base Model=Qwen3-14B, Scaffold=R2EGym, Data=2.2k trajectories2026.04 | 30.06 | |
| LoopCoder-v2R (number of refinement loops)=32026.06 | 27.6 | |
| R2EGymBase Model=Qwen2.5-Coder-14B, Scaffold=R2EGym, Data=3.3k trajectories2026.04 | 26.8 | |
| Best DGM agentAgent Stage=Best, Evaluation Protocol=Transfer (from Polyglot)2026.06 | 24.5 | |
| SWE-AGILE (SFT+RL)Base Model=Qwen3-8B, Scaffold=R2EGym, Data=+ 896 SWE tasks2026.04 | 24.1 | |
| SWE-DevBase Model=Qwen2.5-Coder-7B, Scaffold=OpenHands, Data=19.3k trajectories2026.04 | 23.4 | |
| LoopCoder-v2R (number of refinement loops)=42026.06 | 22.4 | |
| SkyRL-Agent-v0Base Model=Qwen3-14B, Scaffold=OpenHands, Data=-2026.04 | 21.6 | |
| SWE-AGILE (SFT)Base Model=Qwen3-8B, Scaffold=R2EGym, Data=2.2k trajectories2026.04 | 21.45 | |
| SWE-GymBase Model=Qwen2.5-Coder-32B, Scaffold=OpenHands, Data=-2026.04 | 20.6 | |
| Initial agentAgent Stage=Initial, Evaluation Protocol=Endpoint2026.06 | 20 | |
| Initial agentAgent Stage=Initial, Evaluation Protocol=Transfer (from Polyglot)2026.06 | 20 | |
| R2EGymBase Model=Qwen2.5-Coder-7B, Scaffold=R2EGym, Data=3.3k trajectories2026.04 | 19 | |
| SWE-GymBase Model=Qwen2.5-Coder-14B, Scaffold=OpenHands, Data=-2026.04 | 16.4 | |
| Qwen3Base Model=Qwen3-8B, Scaffold=R2EGym, Data=-2026.04 | 15.83 | |
| SWE-smithBase Model=Qwen2.5-Coder-7B, Scaffold=SWE-agent, Data=5k trajectories2026.04 | 15.2 | |
| SWE-GymBase Model=Qwen2.5-Coder-7B, Scaffold=OpenHands, Data=491 trajectories2026.04 | 10.6 | |
| DeepSeek-Coder-V2-Lite-Instruct2026.06 | 0 | |
| Qwen2.5-Coder-7B-Instruct2026.06 | 0 | |
| Seed-Coder-8B-Instruct2026.06 | 0 | |
| Qwen2.5-Coder-14B-Instruct2026.06 | 0 | |
| Qwen2.5-Coder-32B-Instruct2026.06 | 0 |