Repo-level Method Body Completion on RAMBO's benchmark (test)
51.21BLEURAMBO Oracle
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| RAMBO OracleBackbone=GPT3.5 - Turbo (175B)2024.09 | 51.21 | 53.91 | 60.85 | 19.15 | |
| RAMBO OracleBackbone=DeepSeek Coder (6.7B)2024.09 | 50.87 | 53.96 | 63.4 | 18.72 | |
| RAMBOBackbone=DeepSeek Coder (6.7B)2024.09 | 50.44 | 53.2 | 61.7 | 18.94 | |
| RepoCoder OracleBackbone=DeepSeek Coder (6.7B)2024.09 | 50.04 | 52.48 | 58.51 | 17.02 | |
| RAMBOBackbone=GPT3.5 - Turbo (175B)2024.09 | 49.58 | 52.71 | 58.94 | 19.15 | |
| RepoCoder OracleBackbone=GPT3.5 - Turbo (175B)2024.09 | 47.82 | 51.43 | 62.13 | 17.02 | |
| RawRAGBackbone=DeepSeek Coder (6.7B)2024.09 | 46.92 | 50.97 | 60.21 | 15.32 | |
| RepoCoder OracleBackbone=CodeLlama (7B)2024.09 | 46.59 | 49.53 | 55.96 | 14.68 | |
| RepoCoderBackbone=DeepSeek Coder (6.7B)2024.09 | 46.22 | 50.31 | 57.23 | 15.53 | |
| RLCoderBackbone=GPT3.5 - Turbo (175B)2024.09 | 45.12 | 49.48 | 58.72 | 15.32 | |
| RLCoderBackbone=DeepSeek Coder (6.7B)2024.09 | 45.08 | 48.58 | 55.96 | 14.26 | |
| RepoCoderBackbone=GPT3.5 - Turbo (175B)2024.09 | 44.41 | 49.64 | 58.09 | 15.74 | |
| RAMBOBackbone=CodeLlama (7B)2024.09 | 43.16 | 47.76 | 55.96 | 13.62 | |
| RepoCoderBackbone=CodeLlama (7B)2024.09 | 42.98 | 47.66 | 55.32 | 12.77 | |
| RawRAGBackbone=GPT3.5 - Turbo (175B)2024.09 | 42.9 | 45.27 | 55.53 | 12.55 | |
| RAMBO OracleBackbone=CodeLlama (7B)2024.09 | 41.83 | 46.22 | 51.28 | 14.89 | |
| RawRAGBackbone=CodeLlama (7B)2024.09 | 41.57 | 45.96 | 54.89 | 12.55 | |
| RLCoderBackbone=CodeLlama (7B)2024.09 | 39.92 | 44.3 | 54.47 | 10.85 | |
| In-FileBackbone=GPT3.5 - Turbo (175B)2024.09 | 33.93 | 39.25 | 53.11 | 6.38 | |
| In-FileBackbone=DeepSeek Coder (6.7B)2024.09 | 25.99 | 33.84 | 45.11 | 4.04 | |
| In-FileBackbone=CodeLlama (7B)2024.09 | 25.09 | 33.85 | 48.09 | 3.83 |