Code Infilling on HumanEval single-line infilling (test)
87.4Pass Ratecode-davinci-002
Evaluation results
| Method | Links | ||
|---|---|---|---|
| code-davinci-002Inference=Infilling2022.04 | 87.4 | 69.6 | |
| code-davinci-002Inference=Left-to-right reranking2022.04 | 71.8 | 52 | |
| INCODER-6.7BInference=Infilling2022.04 | 69 | 56.3 | |
| code-davinci-002Inference=Left-to-right single2022.04 | 63.7 | 48.4 | |
| INCODER-6.7BInference=Left-to-right reranking2022.04 | 54.9 | 44.1 | |
| INCODER-6.7BInference=Left-to-right single2022.04 | 48.2 | 38.7 | |
| CM infillingsampling temperature=0.2, mode=causal-masked2022.04 | 0.69 | 0.563 | |
| code-davinci-001mode=zero-shot, prompting=templated2022.04 | 0.63 | 0.56 | |
| L-R rerankingsampling temperature=0.8, K=10, mode=left-to-right reranking2022.04 | 0.549 | 0.441 | |
| code-cushman-001mode=zero-shot, prompting=templated2022.04 | 0.531 | 0.42 | |
| L-R singlesampling temperature=0.2, mode=left-to-right2022.04 | 0.482 | 0.387 | |
| PLBARTtype=encoder-decoder, objective=denoising infilling, mode=zero-shot2022.04 | 0.416 | — |