Repository-level Method Body Completion on Defect4J
67.8BLEURAMBO Oracle
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| RAMBO OracleCode LLM=GPT3.5 - Turbo (175B)2024.09 | 67.8 | 67.96 | 76.96 | 61.76 | 35.29 | |
| RAMBOCode LLM=GPT3.5 - Turbo (175B)2024.09 | 66.29 | 67.82 | 76.47 | 63.73 | 32.84 | |
| RawRAGCode LLM=GPT3.5 - Turbo (175B)2024.09 | 66.09 | 67.36 | 75 | 61.27 | 31.28 | |
| RLCoderCode LLM=GPT3.5 - Turbo (175B)2024.09 | 65.78 | 66.67 | 75.98 | 61.76 | 32.84 | |
| RepoCoderCode LLM=GPT3.5 - Turbo (175B)2024.09 | 63.52 | 65.39 | 74.02 | 59.8 | 31.86 | |
| In-FileCode LLM=GPT3.5 - Turbo (175B)2024.09 | 63.16 | 65.66 | 76.96 | 60.78 | 26.96 | |
| RepoCoder OracleCode LLM=GPT3.5 - Turbo (175B)2024.09 | 62.76 | 64.85 | 72.06 | 57.84 | 31.37 | |
| RAMBO OracleCode LLM=CodeLlama (7B)2024.09 | 54.19 | 55.9 | 64.71 | 46.08 | 19.12 | |
| RepoCoder OracleCode LLM=CodeLlama (7B)2024.09 | 53.69 | 54.47 | 64.71 | 46.57 | 19.61 | |
| RAMBOCode LLM=CodeLlama (7B)2024.09 | 52.04 | 54.98 | 63.73 | 44.12 | 20.59 | |
| RAMBO OracleCode LLM=DeepSeek Coder (6.7B)2024.09 | 51.31 | 53.54 | 67.16 | 49.51 | 21.57 | |
| RepoCoderCode LLM=CodeLlama (7B)2024.09 | 51.27 | 53.46 | 60.78 | 46.57 | 20.1 | |
| RAMBOCode LLM=DeepSeek Coder (6.7B)2024.09 | 51.03 | 53.05 | 69.61 | 46.57 | 23.04 | |
| RepoCoder OracleCode LLM=DeepSeek Coder (6.7B)2024.09 | 50.74 | 52.96 | 64.71 | 45.59 | 21.57 | |
| RLCoderCode LLM=CodeLlama (7B)2024.09 | 50.37 | 51.12 | 60.29 | 42.65 | 17.54 | |
| RawRAGCode LLM=CodeLlama (7B)2024.09 | 48.88 | 50.93 | 60.29 | 45.1 | 18.01 | |
| RepoCoderCode LLM=DeepSeek Coder (6.7B)2024.09 | 48.39 | 51.01 | 60.78 | 46.08 | 20.59 | |
| RLCoderCode LLM=DeepSeek Coder (6.7B)2024.09 | 45.4 | 48.1 | 57.84 | 45.59 | 18.01 | |
| RawRAGCode LLM=DeepSeek Coder (6.7B)2024.09 | 44.96 | 48.47 | 61.27 | 45.1 | 18.01 | |
| In-FileCode LLM=CodeLlama (7B)2024.09 | 44.06 | 46.73 | 55.88 | 33.82 | 8.82 | |
| In-FileCode LLM=DeepSeek Coder (6.7B)2024.09 | 39.18 | 43.24 | 52.45 | 34.8 | 11.27 |