Automated Program Repair on SWE-bench Verified 500 instances
41Pass@1 RateSWE-RL
Evaluation results
| Method | Links | |
|---|---|---|
| SWE-RLBackbone=Llama-3-70B, Params=70B2026.05 | 41 | |
| BOOSTAPR (+ Rline)Params=32B2026.05 | 40.7 | |
| AgentlessBackbone=GPT-4o2026.05 | 38.8 | |
| + Stage III (PPO, Rseq only)Params=32B2026.05 | 38.3 | |
| SWE-agentBackbone=Claude 3.5 Sonnet2026.05 | 33.6 | |
| SWE-FixerBackbone=Qwen2.5-72B, Params=72B2026.05 | 33 | |
| SWE-GymBackbone=Qwen2.5-Coder-32B, Params=32B2026.05 | 32 | |
| Lingma SWE-GPTBackbone=Qwen2.5-72B, Params=72B2026.05 | 30.2 | |
| AutoCodeRoverBackbone=GPT-4o2026.05 | 28.8 | |
| + Stage I (SFT)Params=32B2026.05 | 23.4 | |
| ChatRepairBackbone=GPT-3.5-turbo2026.05 | 18.2 | |
| Qwen2.5-Coder-32B (base)Params=32B2026.05 | 17.8 | |
| RLEFBackbone=Llama-3-8B, Params=8B2026.05 | 12.6 | |
| RepairLLaMABackbone=CodeLlama-7B, Params=7B2026.05 | 8.6 | |
| CodeRLBackbone=CodeT5-large, Params=770M2026.05 | 3.2 | |
| KNODBackbone=CodeT5-base, Params=220M2026.05 | 2.4 |