Code Generation on Python deduplicated retrieval codebase (test)
1,183EMCodeT5+
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CodeT5+Evaluation Setting=Retrieval-Augmented Generative, Parameters=220M2023.05 | 1,183 | 31.14 | 36.39 | |
| REDCODER-EXTEvaluation Setting=Retrieval-Augmented Generative, Parameters=125M+140M2023.05 | 961 | 24.43 | 30.21 | |
| BM25Evaluation Setting=Retrieval-based2023.05 | 0 | 6.63 | 13.49 | |
| SCODE-REvaluation Setting=Retrieval-based, Parameters=125M2023.05 | 0 | 22.75 | 23.92 | |
| CodeT5+Evaluation Setting=Retrieval-based, Parameters=220M2023.05 | 0 | 27.3 | 26.51 | |
| CodeBERTEvaluation Setting=Generative, Parameters=125M2023.05 | 0 | 4.06 | 10.42 | |
| GraphCodeBERTEvaluation Setting=Generative, Parameters=125M2023.05 | 0 | 3.97 | 10.55 | |
| PLBARTEvaluation Setting=Generative, Parameters=140M2023.05 | 0 | 4.89 | 12.01 | |
| CodeT5+Evaluation Setting=Generative, Parameters=220M2023.05 | 0 | 4.4 | 13.88 |