Software Engineering on SWE Verified
77.2Resolution RateClaude-4.5-Sonnet
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Claude-4.5-Sonnet2025.12 | 77.2 | — | |
| Gemini-3.0 Pro2025.12 | 76.2 | — | |
| GPT-5 High2025.12 | 74.9 | — | |
| DeepSeek-V3.2thinking mode=true2025.12 | 73.1 | — | |
| Kimi-K2thinking mode=true2025.12 | 71.3 | — | |
| MiniMax M22025.12 | 69.4 | — | |
| Claude-3.5-SonnetFramework=Openhands, Type=Agent, Model Category=Proprietary Models2025.02 | 53 | — | |
| Claude-3.5-SonnetFramework=Agentless, Type=Pipeline, Model Category=Proprietary Models2025.02 | 50.8 | — | |
| GPT-4oFramework=SWE-SynInfer, Type=Pipeline + Agent, Model Category=Proprietary Models2025.02 | 31.8 | — | |
| SoRFT-Qwen-32BFramework=Agentless, Type=Pipeline, Model Category=32-72B Open-source Models2025.02 | 30.8 | — | |
| Lingma-SWE-GPT-72BFramework=SWE-SynInfer, Type=Pipeline + Agent, Model Category=32-72B Open-source Models2025.02 | 30.2 | — | |
| SWE-Fixer-Qwen-72BFramework=SWE-Fixer, Type=Pipeline, Model Category=32-72B Open-source Models2025.02 | 30.2 | — | |
| SoRFT-Qwen-7BFramework=Agentless, Type=Pipeline, Model Category=7-14B Open-source Models2025.02 | 21.4 | — | |
| SWE-Gym-Qwen-32BFramework=Openhands, Type=Agent, Model Category=32-72B Open-source Models2025.02 | 20.6 | — | |
| Lingma-SWE-GPT-7BFramework=SWE-SynInfer, Type=Pipeline + Agent, Model Category=7-14B Open-source Models2025.02 | 18.2 | — | |
| SWE-Gym-Qwen-14BFramework=Openhands, Type=Agent, Model Category=7-14B Open-source Models2025.02 | 16.4 | — | |
| SWE-Gym-Qwen-7BFramework=Openhands, Type=Agent, Model Category=7-14B Open-source Models2025.02 | 10.6 | — | |
| Claude-3.5-Sonnet-10222025.01 | — | 50.8 | |
| DeepSeek-R1Architecture=MoE, Activated Params=37B, Total Params=671B2025.01 | — | 49.2 | |
| DeepSeek-V3Architecture=MoE, Activated Params=37B, Total Params=671B2025.01 | — | 42 | |
| GPT-4o-05132025.01 | — | 38.8 | |
| OpenAI-o1-12172025.01 | — | 48.9 | |
| OpenAI-o1-mini2025.01 | — | 41.6 |