Docstring Generation on CodeXGLUE Python (test)
21.99BLEUStarCoder
Evaluation results
| Method | Links | ||
|---|---|---|---|
| StarCodermode=zero-shot, capability=infilling, decoding=greedy2023.05 | 21.99 | — | |
| StarCoderBasemode=zero-shot, capability=infilling, decoding=greedy2023.05 | 21.38 | — | |
| CodeT5Evaluation Protocol=Finetuned2022.04 | 20.36 | — | |
| SantaCodermode=zero-shot, capability=infilling, decoding=greedy2023.05 | 19.74 | — | |
| PLBARTEvaluation Protocol=Finetuned2022.04 | 19.3 | — | |
| CodeBERTEvaluation Protocol=Finetuned2022.04 | 19.06 | — | |
| InCoderEvaluation Protocol=Zero-shot, Inference Strategy=Causal-masked infilling2022.04 | 18.27 | — | |
| InCoder-6Bmode=zero-shot, capability=infilling, decoding=greedy2023.05 | 18.27 | — | |
| RoBERTaEvaluation Protocol=Finetuned2022.04 | 18.14 | — | |
| InCoderEvaluation Protocol=Zero-shot, Inference Strategy=L-R reranking2022.04 | 17.14 | — | |
| InCoderEvaluation Protocol=Zero-shot, Inference Strategy=L-R single2022.04 | 16.05 | — | |
| Code LlamaSize=7B, LCFT=false, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 20.39 | |
| Code LlamaSize=7B, LCFT=true, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 20.37 | |
| Code LlamaSize=13B, LCFT=false, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 21.05 | |
| Code LlamaSize=13B, LCFT=true, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 21.15 | |
| InCoderSize=6B, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 18.27 | |
| SantaCoderSize=1.1B, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 19.74 | |
| StarCoderSize=15.5B, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 21.99 | |
| StarCoderBaseSize=15.5B, Decoding Format=PSM, Decoding Strategy=greedy2023.08 | — | 21.38 |