Line Completion on RepoEval 1.0 (test)
57.75Exact MatchOracle
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OracleBackbone=GPT-3.5-Turbo2023.03 | 57.75 | 75.43 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=32023.03 | 57 | 75.3 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=22023.03 | 56.81 | 75.11 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=42023.03 | 56.63 | 75.1 | |
| RepoCoderBackbone=GPT-3.5-Turbo, Iterations=12023.03 | 55.31 | 74.38 | |
| OracleBackbone=CODEGEN-MONO-6B2023.03 | 48.81 | 71.02 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=32023.03 | 47.75 | 70.73 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=42023.03 | 47.44 | 70.19 | |
| OracleBackbone=CODEGEN-MONO-2B2023.03 | 47.31 | 69.8 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=42023.03 | 47.13 | 68.92 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=22023.03 | 47.06 | 70.1 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=22023.03 | 46.94 | 68.82 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=32023.03 | 46.69 | 68.62 | |
| RepoCoderBackbone=CODEGEN-MONO-6B, Iterations=12023.03 | 45.81 | 69.21 | |
| OracleBackbone=CODEGEN-MONO-350M2023.03 | 45.19 | 67.2 | |
| RepoCoderBackbone=CODEGEN-MONO-2B, Iterations=12023.03 | 44.56 | 67.68 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=32023.03 | 43.94 | 65.97 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=22023.03 | 43.06 | 65.66 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=42023.03 | 43.06 | 65.62 | |
| RepoCoderBackbone=CODEGEN-MONO-350M, Iterations=12023.03 | 41.88 | 65.05 | |
| In-FileBackbone=GPT-3.5-Turbo2023.03 | 40.56 | 65.06 | |
| In-FileBackbone=CODEGEN-MONO-6B2023.03 | 34.56 | 60.67 | |
| In-FileBackbone=CODEGEN-MONO-2B2023.03 | 33.63 | 58.99 | |
| In-FileBackbone=CODEGEN-MONO-350M2023.03 | 29.56 | 55.39 |