# Automated Program Repair on SWE-bench Verified 500 instances Source: [Automated Program Repair on SWE-bench Verified 500 instances]() Collection index: [llms.txt]() Data updated: 2026-05-12T11:21:31.341Z Evaluation of model performance on automatically repairing software bugs using the SWE-bench Verified 500 instances dataset, measured by pass@1 success rate. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Automated Program Repair | | Dataset (including any stated split) | SWE-bench Verified 500 instances | | Metric | Pass@1 Rate | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1 Rate](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 Rate | Result date | Report paper | | --- | --- | --- | --- | | SWE-RL | 41 | Not recorded | 6a02b45cdbc836b87d8249ed | | BOOSTAPR (+ Rline) | 40.7 | Not recorded | 6a02b45cdbc836b87d8249ed | | Agentless | 38.8 | Not recorded | 6a02b45cdbc836b87d8249ed | | + Stage III (PPO, Rseq only) | 38.3 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-agent | 33.6 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-Fixer | 33 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-Gym | 32 | Not recorded | 6a02b45cdbc836b87d8249ed | | Lingma SWE-GPT | 30.2 | Not recorded | 6a02b45cdbc836b87d8249ed | | AutoCodeRover | 28.8 | Not recorded | 6a02b45cdbc836b87d8249ed | | + Stage I (SFT) | 23.4 | Not recorded | 6a02b45cdbc836b87d8249ed | | ChatRepair | 18.2 | Not recorded | 6a02b45cdbc836b87d8249ed | | Qwen2.5-Coder-32B (base) | 17.8 | Not recorded | 6a02b45cdbc836b87d8249ed | | RLEF | 12.6 | Not recorded | 6a02b45cdbc836b87d8249ed | | RepairLLaMA | 8.6 | Not recorded | 6a02b45cdbc836b87d8249ed | | CodeRL | 3.2 | Not recorded | 6a02b45cdbc836b87d8249ed | | KNOD | 2.4 | Not recorded | 6a02b45cdbc836b87d8249ed | ### SWE-RL Pass@1 Rate: 41. Result ID: 6a030cba47f34ce924e6cd9d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-70B","Params":"70B"}. This record does not assert independent reproduction. ### BOOSTAPR (+ Rline) Pass@1 Rate: 40.7. Result ID: 6a030cba47f34ce924e6cda1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### Agentless Pass@1 Rate: 38.8. Result ID: 6a030cba47f34ce924e6cd92. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o","Params":null}. This record does not assert independent reproduction. ### + Stage III (PPO, Rseq only) Pass@1 Rate: 38.3. Result ID: 6a030cba47f34ce924e6cda0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### SWE-agent Pass@1 Rate: 33.6. Result ID: 6a030cba47f34ce924e6cd93. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Claude 3.5 Sonnet","Params":null}. This record does not assert independent reproduction. ### SWE-Fixer Pass@1 Rate: 33. Result ID: 6a030cba47f34ce924e6cd98. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-72B","Params":"72B"}. This record does not assert independent reproduction. ### SWE-Gym Pass@1 Rate: 32. Result ID: 6a030cba47f34ce924e6cd96. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-32B","Params":"32B"}. This record does not assert independent reproduction. ### Lingma SWE-GPT Pass@1 Rate: 30.2. Result ID: 6a030cba47f34ce924e6cd97. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-72B","Params":"72B"}. This record does not assert independent reproduction. ### AutoCodeRover Pass@1 Rate: 28.8. Result ID: 6a030cba47f34ce924e6cd94. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o","Params":null}. This record does not assert independent reproduction. ### + Stage I (SFT) Pass@1 Rate: 23.4. Result ID: 6a030cba47f34ce924e6cd9f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### ChatRepair Pass@1 Rate: 18.2. Result ID: 6a030cba47f34ce924e6cd95. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5-turbo","Params":null}. This record does not assert independent reproduction. ### Qwen2.5-Coder-32B (base) Pass@1 Rate: 17.8. Result ID: 6a030cba47f34ce924e6cd9e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### RLEF Pass@1 Rate: 12.6. Result ID: 6a030cba47f34ce924e6cd9c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Params":"8B"}. This record does not assert independent reproduction. ### RepairLLaMA Pass@1 Rate: 8.6. Result ID: 6a030cba47f34ce924e6cd99. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B","Params":"7B"}. This record does not assert independent reproduction. ### CodeRL Pass@1 Rate: 3.2. Result ID: 6a030cba47f34ce924e6cd9b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"CodeT5-large","Params":"770M"}. This record does not assert independent reproduction. ### KNOD Pass@1 Rate: 2.4. Result ID: 6a030cba47f34ce924e6cd9a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeT5-base","Params":"220M"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.