Software Engineering Issue Resolution on SWE-Bench Lite
73.5Resolution RateAIR
Evaluation results
| Method | Links | |
|---|---|---|
| AIRBackbone=Gemini 2.52025.12 | 73.5 | |
| SWE-agentBackbone=Claude 4 Sonnet2025.12 | 56.67 | |
| Claude-3.5-SonnetFramework=Openhands, Type=Agent, Model Category=Proprietary Models2025.02 | 41.7 | |
| Claude-3.5-SonnetFramework=Agentless, Type=Pipeline, Model Category=Proprietary Models2025.02 | 40.7 | |
| AutoCodeRoverBackbone=GPT-4o2025.12 | 37.3 | |
| SWE-agentBackbone=Claude 3.5 Sonnet2025.12 | 33.6 | |
| OpenHandsBackbone=Claude 3.5 Sonnet2025.12 | 27.7 | |
| SoRFT-Qwen-32BFramework=Agentless, Type=Pipeline, Model Category=32-72B Open-source Models2025.02 | 24 | |
| SWE-Fixer-Qwen-72BFramework=SWE-Fixer, Type=Pipeline, Model Category=32-72B Open-source Models2025.02 | 23.3 | |
| Lingma-SWE-GPT-72BFramework=SWE-SynInfer, Type=Pipeline + Agent, Model Category=32-72B Open-source Models2025.02 | 22 | |
| GPT-4oFramework=SWE-SynInfer, Type=Pipeline + Agent, Model Category=Proprietary Models2025.02 | 20.7 | |
| SWE-Gym-Qwen-32BFramework=Openhands, Type=Agent, Model Category=32-72B Open-source Models2025.02 | 15.3 | |
| SoRFT-Qwen-7BFramework=Agentless, Type=Pipeline, Model Category=7-14B Open-source Models2025.02 | 14 | |
| SWE-Gym-Qwen-14BFramework=Openhands, Type=Agent, Model Category=7-14B Open-source Models2025.02 | 12.7 | |
| Lingma-SWE-GPT-7BFramework=SWE-SynInfer, Type=Pipeline + Agent, Model Category=7-14B Open-source Models2025.02 | 12 | |
| SWE-Gym-Qwen-7BFramework=Openhands, Type=Agent, Model Category=7-14B Open-source Models2025.02 | 10 |