Code Infilling on HumanEval multi-line code infilling
38.6Pass RateCM infilling
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CM infillingTemperature=0.22022.04 | 38.6 | 20.6 | — | — | — | |
| CM infillingsampling temperature=0.2, mode=causal-masked2022.04 | 38.6 | 20.6 | — | — | — | |
| code-davinci-001Zero-shot=true2022.04 | 37.8 | 19.8 | — | — | — | |
| code-davinci-001mode=zero-shot, prompting=templated2022.04 | 37.8 | 19.8 | — | — | — | |
| code-cushman-001Zero-shot=true2022.04 | 30.8 | 17.4 | — | — | — | |
| code-cushman-001mode=zero-shot, prompting=templated2022.04 | 30.8 | 17.4 | — | — | — | |
| L-R rerankingTemperature=0.8, K=102022.04 | 28.2 | 17.6 | — | — | — | |
| L-R rerankingsampling temperature=0.8, K=10, mode=left-to-right reranking2022.04 | 28.2 | 17.6 | — | — | — | |
| L-R singleTemperature=0.22022.04 | 24.9 | 15.8 | — | — | — | |
| L-R singlesampling temperature=0.2, mode=left-to-right2022.04 | 24.9 | 15.8 | — | — | — | |
| PLBARTZero-shot=true2022.04 | 13.1 | — | — | — | — | |
| PLBARTtype=encoder-decoder, objective=denoising infilling, mode=zero-shot2022.04 | 13.1 | — | — | — | — | |
| Code LlamaSize=7B, LCFT=false, Decoding Strategy=greedy decoding2023.08 | — | — | 49.7 | 51.2 | — | |
| Code LlamaSize=7B, LCFT=true, Decoding Strategy=greedy decoding2023.08 | — | — | 48.2 | 50.8 | — | |
| Code LlamaSize=13B, LCFT=false, Decoding Strategy=greedy decoding2023.08 | — | — | 53.7 | 56.7 | — | |
| Code LlamaSize=13B, LCFT=true, Decoding Strategy=greedy decoding2023.08 | — | — | 51 | 56.1 | — | |
| code-davinci-002Size=175B, Temperature=0.1, Decoding Strategy=nucleus sampling (p=0.95)2023.08 | — | — | 69.9 | — | — | |
| Deepseek-Coder-6.7BModel Type=Open-Weights AR2026.02 | — | — | — | — | 45.7 | |
| DiffuCoder-7BModel Type=Open-Weights Diffusion2026.02 | — | — | — | — | 45 | |
| Dream-7BModel Type=Open-Weights Diffusion2026.02 | — | — | — | — | 21.9 | |
| DreamCoder-7BModel Type=Open-Weights Diffusion2026.02 | — | — | — | — | 43.2 | |
| DREAMONModel Type=Open-Weights Diffusion, Backbone=Dream-7B2026.02 | — | — | — | — | 53.3 | |
| DREAMONModel Type=Open-Weights Diffusion, Backbone=DiffuCoder-7B2026.02 | — | — | — | — | 63.1 | |
| DREAMONModel Type=Open-Weights Diffusion, Backbone=DreamCoder-7B2026.02 | — | — | — | — | 63.8 | |
| InCoderSize=6B, Temperature=0.2, Decoding Strategy=sampling2023.08 | — | — | 38.6 | — | — | |
| LLaDA-8BModel Type=Open-Weights Diffusion2026.02 | — | — | — | — | 21.1 | |
| OpenAI FIM90Size=7B, Temperature=0.1, Decoding Strategy=nucleus sampling (p=0.95)2023.08 | — | — | 44.1 | — | — | |
| Qwen2.5-Coder-7BModel Type=Open-Weights AR2026.02 | — | — | — | — | 58.7 | |
| Seed-Coder-8BModel Type=Open-Weights AR2026.02 | — | — | — | — | 59.3 |