Code Generation on HumanEval 1.0 (test)
85.4Pass@1GPT4-Turbo
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GPT4-Turbo2023.12 | 85.4 | — | — | — | — | |
| IF-CLSize=34B, Open-source=weight&data, Base Model=CODELLAMA2023.12 | 78.7 | — | — | — | — | |
| IF-CLPSize=34B, Open-source=weight&data, Base Model=CODELLAMA-PYTHON2023.12 | 75.6 | — | — | — | — | |
| IF-CLSize=13B, Open-source=weight&data, Base Model=CODELLAMA2023.12 | 74.4 | — | — | — | — | |
| IF-CLPSize=13B, Open-source=weight&data, Base Model=CODELLAMA-PYTHON2023.12 | 73.8 | — | — | — | — | |
| WizardCoder-CLPSize=34B, Open-source=weight2023.12 | 73.2 | — | — | — | — | |
| GPT3.5-Turbo2023.12 | 72.6 | — | — | — | — | |
| MagicoderS-CLPSize=7B, Open-source=weight&data2023.12 | 70.7 | — | — | — | — | |
| GPT-4Category=Closed-source models2023.05 | 67 | — | — | — | — | |
| CODELLAMA-PYTHONSize=34B, Open-source=weight2023.12 | 51.8 | — | — | — | — | |
| WizardCoder-CLPSize=7B, Open-source=weight2023.12 | 48.2 | — | — | — | — | |
| GPT-3.5Category=Closed-source models2023.05 | 48.1 | — | — | — | — | |
| code-davinci-002Category=Closed-source models2023.05 | 47 | 74.9 | 92.1 | — | — | |
| InstructCodeT5+ w/ CodeTModel size=16B, Generation strategy=CodeT, Category=Open-source models + generation strategies2023.05 | 42.9 | 67.8 | 78.7 | — | — | |
| CODELLAMA-PYTHONSize=13B, Open-source=weight2023.12 | 42.7 | — | — | — | — | |
| StarCoder (prompted)Model size=15B, Category=Open-source models + generation strategies2023.05 | 40.8 | — | — | — | — | |
| CodeT5+ w/ CodeTModel size=16B, Generation strategy=CodeT, Category=Open-source models + generation strategies2023.05 | 38.5 | 63.6 | 77.1 | — | — | |
| CODELLAMA-PYTHONSize=7B, Open-source=weight2023.12 | 37.8 | — | — | — | — | |
| CodeGen-mono w/ CodeTModel size=16B, Generation strategy=CodeT, Category=Open-source models + generation strategies2023.05 | 36.7 | 59.3 | — | — | — | |
| PaLM-CoderSize (B)=540, Python Code (GB)=~20, Other Code (GB)=~200, Other (GB)=~4000, Code License=Permissive, Infill?=false2022.04 | 36 | — | 88.4 | — | — | |
| PaLM-CoderModel size=540B, Category=Closed-source models2023.05 | 36 | — | 88.4 | — | — | |
| InstructCodeT5+Model size=16B, Category=Open-source models2023.05 | 35 | 54.5 | 77.9 | — | — | |
| StarCoderSize=15B, Open-source=weight&data2023.12 | 34.1 | — | — | — | — | |
| StarCoderModel size=15B, Category=Open-source models2023.05 | 33.6 | — | — | — | — | |
| code-cushman-001Category=Closed-source models2023.05 | 33.5 | 54.3 | 77.4 | — | — | |
| CodeGen-MonoSize=16B, Open-source=weight&data2023.12 | 32.9 | — | — | — | — | |
| CodeT5+Size=16B, Open-source=weight&data2023.12 | 31.7 | — | — | — | — | |
| CodeT5+Model size=16B, Category=Open-source models2023.05 | 30.9 | 51.6 | 76.7 | — | — | |
| MIMModel size=2.7B, Category=Closed-source models2023.05 | 30.7 | 48.2 | 69.6 | — | — | |
| CodeGen-MonoSize (B)=16.1, Python Code (GB)=279, Other Code (GB)=375, Other (GB)=1200, Infill?=false2022.04 | 29.3 | 49.9 | 75 | — | — | |
| CodeGen-monoModel size=16B, Category=Open-source models2023.05 | 29.3 | 49.9 | 75 | — | — | |
| CODEXParameters=12B2021.07 | 28.81 | 46.81 | 72.31 | — | — | |
| Codex-12BSize (B)=12, Python Code (GB)=180, Other (GB)=> 570, Infill?=false2022.04 | 28.8 | 46.8 | 72.3 | — | — | |
| CodexModel size=12B, Category=Closed-source models2023.05 | 28.8 | 46.8 | 72.3 | — | — | |
| MistralSize=7B, Open-source=weight2023.12 | 28.7 | — | — | — | — | |
| CodeT5+Model size=6B, Category=Open-source models2023.05 | 28 | 47.2 | 69.8 | — | — | |
| PaLMModel size=540B, Category=Closed-source models2023.05 | 26.2 | — | 76.2 | — | — | |
| CodeGen-MonoSize (B)=6.1, Python Code (GB)=279, Other Code (GB)=375, Other (GB)=1200, Infill?=false2022.04 | 26.1 | 42.3 | 65.8 | — | — | |
| CodeGen-monoModel size=6B, Category=Open-source models2023.05 | 26.1 | 42.3 | 65.8 | — | — | |
| StarCoderSize=7B, Open-source=weight&data2023.12 | 24.4 | — | — | — | — | |
| CodeT5+Model size=2B, Category=Open-source models2023.05 | 24.2 | 38.2 | 57.8 | — | — | |
| LLaMAModel size=65B, Category=Open-source models2023.05 | 23.7 | — | 79.3 | — | — | |
| CodeGen-monoModel size=2B, Category=Open-source models2023.05 | 23.7 | 36.6 | 57 | — | — | |
| CodeGeeXModel size=13B, Category=Open-source models2023.05 | 22.9 | 39.6 | 60.9 | — | — | |
| MIMModel size=1.3B, Category=Closed-source models2023.05 | 22.4 | 41.7 | 53.8 | — | — | |
| ReplitModel size=3B, Category=Open-source models2023.05 | 21.9 | — | — | — | — | |
| LLAMAModel size=33B, Category=Open-source models2023.05 | 21.7 | — | 70.7 | — | — | |
| Codex-2.5BSize (B)=2.5, Python Code (GB)=180, Other (GB)=> 570, Infill?=false2022.04 | 21.4 | 35.4 | 59.5 | — | — | |
| CodexModel size=2.5B, Category=Closed-source models2023.05 | 21.4 | 35.4 | 59.5 | — | — | |
| CODEXParameters=2.5B2021.07 | 21.36 | 35.42 | 59.5 | — | — | |
| CodeGen-MultiSize (B)=6.1, Python Code (GB)=62, Other Code (GB)=375, Other (GB)=1200, Infill?=false2022.04 | 18.2 | 28.7 | 44.9 | — | — | |
| AlphaCodeSize (B)=1.1, Python Code (GB)=54, Other Code (GB)=660, Infill?=false2022.04 | 17.1 | 28.2 | 45.3 | — | — | |
| AlphaCodeModel size=1.1B, Category=Closed-source models2023.05 | 17.1 | 28.2 | 45.3 | — | — | |
| CODEXParameters=679M2021.07 | 16.22 | 25.7 | 40.95 | — | — | |
| PaLMModel size=62B, Category=Closed-source models2023.05 | 15.9 | — | 46.3 | — | — | |
| LLAMAModel size=13B, Category=Open-source models2023.05 | 15.8 | — | 52.5 | — | — | |
| CodeT5+Model size=770M, Category=Open-source models2023.05 | 15.5 | 27.2 | 42.7 | — | — | |
| GPT-NeoXSize (B)=20, Python Code (GB)=6, Other Code (GB)=90, Other (GB)=730, Infill?=false2022.04 | 15.4 | 25.6 | 41.2 | — | — | |
| GPT-NeoXModel size=20B, Category=Open-source models2023.05 | 15.4 | 25.6 | 41.2 | — | — | |
| INCODER-6.7BSize (B)=6.7, Python Code (GB)=52, Other Code (GB)=107, Other (GB)=57, Code License=Permissive, Infill?=true2022.04 | 15.2 | 27.8 | 47 | — | — | |
| InCoderModel size=6B, Category=Open-source models2023.05 | 15.2 | 27.8 | 47 | — | — | |
| LaMDASize (B)=137, Infill?=false2022.04 | 14 | — | 47.3 | — | — | |
| LaMDAModel size=137B, Category=Closed-source models2023.05 | 14 | — | 47.3 | — | — | |
| CODEXParameters=300M2021.07 | 13.17 | 20.37 | 36.27 | — | — | |
| CodeT5+Model size=220M, Category=Open-source models2023.05 | 12 | 20.7 | 31.6 | — | — | |
| GPT-JParameters=6B2021.07 | 11.62 | 15.74 | 27.74 | — | — | |
| GPT-JSize (B)=6, Python Code (GB)=6, Other Code (GB)=90, Other (GB)=730, Infill?=false2022.04 | 11.6 | 15.7 | 27.7 | — | — | |
| GPT-JModel size=6B, Category=Open-source models2023.05 | 11.6 | 15.7 | — | — | — | |
| LLAMAModel size=7B, Category=Open-source models2023.05 | 10.5 | — | 36.5 | — | — | |
| InCoderModel size=1.3B, Category=Open-source models2023.05 | 8.9 | 16.7 | 25.6 | — | — | |
| CODEXParameters=85M2021.07 | 8.22 | 12.81 | 22.4 | — | — | |
| GPT-NEOParameters=2.7B2021.07 | 6.41 | 11.27 | 21.37 | — | — | |
| GPT-NeoModel size=2.7B, Category=Open-source models2023.05 | 6.4 | 11.3 | 21.4 | — | — | |
| PolyCoderSize (B)=2.7, Python Code (GB)=16, Other Code (GB)=238, Infill?=false2022.04 | 5.6 | 9.8 | 17.7 | — | — | |
| CODEXParameters=42M2021.07 | 5.06 | 8.8 | 15.55 | — | — | |
| GPT-NEOParameters=1.3B2021.07 | 4.79 | 9.47 | 16.3 | — | — | |
| CodeParrotSize (B)=1.5, Python Code (GB)=50, Infill?=false2022.04 | 4 | 8.7 | 17.9 | — | — | |
| PaLMModel size=8B, Category=Closed-source models2023.05 | 3.6 | — | 18.7 | — | — | |
| CODEXParameters=25M2021.07 | 3.21 | 7.1 | 12.89 | — | — | |
| TABNINE2021.07 | 2.58 | 4.35 | 7.59 | — | — | |
| CODEXParameters=12M2021.07 | 2 | 3.62 | 8.58 | — | — | |
| Focused-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 0.927 | — | — | 1.29 | — | |
| Token-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 0.927 | — | — | — | — | |
| SFTBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 0.927 | — | — | — | — | |
| DPO/Step-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 0.921 | — | — | — | — | |
| Qwen2.5-coder-instruct-7BModel Type=Instruct2025.02 | 0.915 | — | — | — | — | |
| Focused-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 0.884 | — | — | 5.89 | — | |
| Token-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 0.866 | — | — | — | — | |
| DPO/Step-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 0.848 | — | — | — | — | |
| SFTBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 0.848 | — | — | — | — | |
| Qwen2.5-coder-baseModel Type=Base2025.02 | 0.835 | — | — | — | — | |
| LDBModel=GPT-3.5 (>=175B), Debugger=LDB2024.02 | 0.829 | — | — | 9.1 | — | |
| Focused-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 0.823 | — | — | 6.35 | — | |
| Focused-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 0.823 | — | — | 12.5 | — | |
| SD (+Expl.)Model=GPT-3.5 (>=175B), Debugger=SD (+Expl.)2024.02 | 0.811 | — | — | 7.3 | — | |
| Token-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 0.811 | — | — | — | — | |
| SD (+Trace)Model=GPT-3.5 (>=175B), Debugger=SD (+Trace)2024.02 | 0.805 | — | — | 6.7 | — | |
| Token-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 0.799 | — | — | — | — | |
| DPO/Step-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 0.787 | — | — | — | — | |
| SFTBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 0.787 | — | — | — | — |