Automated Software Engineering on SWE-bench Lite
33Resolve RateOEC-SWE-32B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OEC-SWE-32BScaffold=SWE-Agent, Train size=4,961, Evaluation=Own evaluation2025.12 | 33 | — | |
| SWE-smith-LM-32BScaffold=SWE-Agent, Train size=5,016, Evaluation=Reported2025.12 | 30.7 | — | |
| SWE-smith-LM-32BScaffold=SWE-Agent, Train size=5,016, Evaluation=Own evaluation2025.12 | 28.3 | — | |
| CodePilotModel=Qwen3-8B2026.01 | 24.67 | 85.33 | |
| CodePilot (w/o Hybrid Retrieval)Model=Qwen3-8B, Hybrid Retrieval=Disabled2026.01 | 22.33 | — | |
| CodePilot (w/o Self-Refinement)Model=Qwen3-8B, Self-Refinement=Disabled2026.01 | 21.67 | — | |
| CodePilot (w/o Thinking Mode)Model=Qwen3-8B, Thinking Mode=Disabled2026.01 | 21 | — | |
| CodePilot (w/o MCTS)Model=Qwen3-8B, MCTS=Disabled2026.01 | 20.33 | — | |
| AgentlessModel=Qwen3-8B2026.01 | 19.33 | 80.67 | |
| AgentlessModel=Qwen2.5-7B2026.01 | 18 | 79.33 | |
| OEC-SWE-7BScaffold=SWE-Agent, Train size=8,943, Evaluation=Own evaluation2025.12 | 17 | — | |
| SWE-agentModel=Qwen3-8B2026.01 | 16.67 | 74 | |
| AgentlessModel=DeepSeek-7B2026.01 | 15.33 | 76.67 | |
| SWE-Gym-32BScaffold=OpenHands, Train size=4912025.12 | 15.3 | — | |
| Direct Generation (Baseline)Model=Qwen3-8B2026.01 | 14 | — | |
| AgentlessModel=CodeLlama-7B2026.01 | 12.67 | 71.33 | |
| SWE-smith-LM-7BScaffold=SWE-Agent, Train size=5,0162025.12 | 11.7 | — | |
| SWE-Gym-7BScaffold=OpenHands, Train size=4912025.12 | 10 | — | |
| Direct Gen.Model=CodeLlama-7B2026.01 | 8.33 | 62 |