Code generation on Concode CodeXGLUE (test)
23.4EMREDCODER
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| REDCODERMethod Category=Retrieval augmented generative methods2021.08 | 23.4 | 41.6 | 43.4 | |
| REDCODER-EXTMethod Category=Retrieval augmented generative methods2021.08 | 23.3 | 42.5 | 43.4 | |
| BM25+PLBARTMethod Category=Retrieval augmented generative methods2021.08 | 21.4 | 40.2 | 41.8 | |
| CodeGPT-adaptedMethod Category=Generative methods2021.08 | 20.1 | 32.8 | 36 | |
| GraphCodeBERTMethod Category=Generative methods2021.08 | 18.7 | 33.4 | 35.9 | |
| PLBARTMethod Category=Generative methods2021.08 | 18.6 | 36.7 | 38.5 | |
| CodeGPT-2Method Category=Generative methods2021.08 | 18.3 | 28.7 | 32.7 | |
| CodeBERTMethod Category=Generative methods2021.08 | 18 | 28.7 | 31.4 | |
| GPT-2Method Category=Generative methods2021.08 | 17.4 | 25.4 | 29.7 | |
| Iyer et al. (2019)Method Category=Generative methods2021.08 | 12.2 | 26.6 | — | |
| Guo et al. (2019)Method Category=Generative methods2021.08 | 10.1 | 24.4 | 29.5 | |
| Seq2SeqMethod Category=Generative methods2021.08 | 3.1 | 21.3 | 26.4 | |
| BM25Method Category=Retrieval based methods2021.08 | 0 | 20.3 | 23.7 | |
| SCODE-RMethod Category=Retrieval based methods, Initialization=CodeBERT2021.08 | 0 | 32.6 | 36.5 |