# Automated Program Repair on Defects4J 835 bugs v2.0 Source: [Automated Program Repair on Defects4J 835 bugs v2.0]() Collection index: [llms.txt]() Data updated: 2026-05-12T11:21:31.385Z Evaluation of model performance on Automated Program Repair using the Defects4J 835 bugs v2.0 dataset, typically measured by functional correctness (pass@1). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Automated Program Repair | | Dataset (including any stated split) | Defects4J 835 bugs v2.0 | | Metric | Pass@1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 | Result date | Report paper | | --- | --- | --- | --- | | BOOSTAPR (+ Rline) | 24.8 | Not recorded | 6a02b45cdbc836b87d8249ed | | + Stage III (PPO, Rseq only) | 19.2 | Not recorded | 6a02b45cdbc836b87d8249ed | | RepairLLaMA | 17.2 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-RL | 16.8 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-Fixer | 15.2 | Not recorded | 6a02b45cdbc836b87d8249ed | | + Stage I (SFT) | 14.9 | Not recorded | 6a02b45cdbc836b87d8249ed | | Lingma SWE-GPT | 14.5 | Not recorded | 6a02b45cdbc836b87d8249ed | | ChatRepair | 14.4 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-Gym | 13.1 | Not recorded | 6a02b45cdbc836b87d8249ed | | Agentless | 12.4 | Not recorded | 6a02b45cdbc836b87d8249ed | | Qwen2.5-Coder-32B (base) | 11.3 | Not recorded | 6a02b45cdbc836b87d8249ed | | SWE-agent | 10.8 | Not recorded | 6a02b45cdbc836b87d8249ed | | AutoCodeRover | 9.6 | Not recorded | 6a02b45cdbc836b87d8249ed | | RLEF | 8.4 | Not recorded | 6a02b45cdbc836b87d8249ed | | KNOD | 6 | Not recorded | 6a02b45cdbc836b87d8249ed | | CodeRL | 5.8 | Not recorded | 6a02b45cdbc836b87d8249ed | ### BOOSTAPR (+ Rline) Pass@1: 24.8. Result ID: 6a030cba47f34ce924e6cdb1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### + Stage III (PPO, Rseq only) Pass@1: 19.2. Result ID: 6a030cba47f34ce924e6cdb0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### RepairLLaMA Pass@1: 17.2. Result ID: 6a030cba47f34ce924e6cda9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B","Params":"7B"}. This record does not assert independent reproduction. ### SWE-RL Pass@1: 16.8. Result ID: 6a030cba47f34ce924e6cdad. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-70B","Params":"70B"}. This record does not assert independent reproduction. ### SWE-Fixer Pass@1: 15.2. Result ID: 6a030cba47f34ce924e6cda8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-72B","Params":"72B"}. This record does not assert independent reproduction. ### + Stage I (SFT) Pass@1: 14.9. Result ID: 6a030cba47f34ce924e6cdaf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### Lingma SWE-GPT Pass@1: 14.5. Result ID: 6a030cba47f34ce924e6cda7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-72B","Params":"72B"}. This record does not assert independent reproduction. ### ChatRepair Pass@1: 14.4. Result ID: 6a030cba47f34ce924e6cda5. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5-turbo","Params":null}. This record does not assert independent reproduction. ### SWE-Gym Pass@1: 13.1. Result ID: 6a030cba47f34ce924e6cda6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-32B","Params":"32B"}. This record does not assert independent reproduction. ### Agentless Pass@1: 12.4. Result ID: 6a030cba47f34ce924e6cda2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o","Params":null}. This record does not assert independent reproduction. ### Qwen2.5-Coder-32B (base) Pass@1: 11.3. Result ID: 6a030cba47f34ce924e6cdae. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":null,"Params":"32B"}. This record does not assert independent reproduction. ### SWE-agent Pass@1: 10.8. Result ID: 6a030cba47f34ce924e6cda3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Claude 3.5 Sonnet","Params":null}. This record does not assert independent reproduction. ### AutoCodeRover Pass@1: 9.6. Result ID: 6a030cba47f34ce924e6cda4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o","Params":null}. This record does not assert independent reproduction. ### RLEF Pass@1: 8.4. Result ID: 6a030cba47f34ce924e6cdac. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Params":"8B"}. This record does not assert independent reproduction. ### KNOD Pass@1: 6. Result ID: 6a030cba47f34ce924e6cdaa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeT5-base","Params":"220M"}. This record does not assert independent reproduction. ### CodeRL Pass@1: 5.8. Result ID: 6a030cba47f34ce924e6cdab. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"CodeT5-large","Params":"770M"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.