API Invocation Completion on RepoEval 1.0 (test)
50.13Exact MatchGPT-3.5-Turbo
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| GPT-3.5-TurboRetrieval Protocol=Oracle2023.03 | 50.13 | 74.5 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=42023.03 | 49.56 | 74.48 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=32023.03 | 49.44 | 74.59 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=22023.03 | 49.19 | 74.43 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=12023.03 | 47.69 | 73.63 | |
| CODEGEN-MONO-6BRetrieval Protocol=Oracle2023.03 | 40.25 | 67.94 | |
| CODEGEN-MONO-2BRetrieval Protocol=Oracle2023.03 | 39.44 | 66.78 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=42023.03 | 39.31 | 65.9 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=32023.03 | 39.13 | 65.53 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=22023.03 | 38.88 | 65.52 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=32023.03 | 38.44 | 64.53 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=42023.03 | 38.25 | 64.6 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=22023.03 | 37.56 | 64.15 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=12023.03 | 36.69 | 64.2 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=12023.03 | 35.44 | 63.47 | |
| CODEGEN-MONO-350MRetrieval Protocol=Oracle2023.03 | 34.88 | 63.06 | |
| GPT-3.5-TurboRetrieval Protocol=In-File2023.03 | 34.06 | 63.22 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=22023.03 | 33.88 | 61.03 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=42023.03 | 33.81 | 61.06 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=32023.03 | 33.75 | 60.96 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=12023.03 | 31.75 | 59.82 | |
| CODEGEN-MONO-6BRetrieval Protocol=In-File2023.03 | 26.19 | 56.45 | |
| CODEGEN-MONO-2BRetrieval Protocol=In-File2023.03 | 25.44 | 56.88 | |
| CODEGEN-MONO-350MRetrieval Protocol=In-File2023.03 | 22.19 | 52.24 |