Code Generation on Java deduplicated retrieval codebase (test)
11.66EMCodeT5+
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CodeT5+Evaluation Setting=Retrieval-Augmented Generative, Parameters=220M2023.05 | 11.66 | 33.83 | 40.6 | |
| REDCODER-EXTEvaluation Setting=Retrieval-Augmented Generative, Parameters=125M+140M2023.05 | 10.21 | 28.98 | 33.18 | |
| BM25Evaluation Setting=Retrieval-based2023.05 | 0 | 4.9 | 16 | |
| SCODE-REvaluation Setting=Retrieval-based, Parameters=125M2023.05 | 0 | 25.34 | 26.68 | |
| CodeT5+Evaluation Setting=Retrieval-based, Parameters=220M2023.05 | 0 | 28.74 | 31 | |
| CodeBERTEvaluation Setting=Generative, Parameters=125M2023.05 | 0 | 8.38 | 14.52 | |
| GraphCodeBERTEvaluation Setting=Generative, Parameters=125M2023.05 | 0 | 7.86 | 14.53 | |
| PLBARTEvaluation Setting=Generative, Parameters=140M2023.05 | 0 | 10.1 | 14.96 | |
| CodeT5+Evaluation Setting=Generative, Parameters=220M2023.05 | 0 | 10.33 | 20.54 |