Code Generation on HumanEval Multilingual (test)
76.5Average ScoreGPT-4
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4Model Type=Instruction-Tuned Model, Setting=Zero-shot2024.01 | 76.5 | 84.1 | 76.4 | 81.6 | 77.2 | 77.4 | 79.1 | 58.2 | 78 | — | — | — | — | — | |
| DeepSeek-Coder-InstructSize=33B, Model Type=Instruction-Tuned Model, Setting=Zero-shot2024.01 | 69.2 | 79.3 | 68.9 | 73.4 | 72.7 | 67.9 | 74.1 | 43 | 73.9 | — | — | — | — | — | |
| DeepSeek-Coder-InstructSize=6.7B, Model Type=Instruction-Tuned Model, Setting=Zero-shot2024.01 | 66.1 | 78.6 | 63.4 | 68.4 | 68.9 | 67.2 | 72.8 | 36.7 | 72.7 | — | — | — | — | — | |
| GPT-3.5-TurboModel Type=Instruction-Tuned Model, Setting=Zero-shot2024.01 | 64.9 | 76.2 | 63.4 | 69.2 | 60.9 | 69.1 | 70.8 | 42.4 | 67.1 | — | — | — | — | — | |
| DeepSeek-Coder-BaseSize=33B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 50.3 | 56.1 | 58.4 | 51.9 | 44.1 | 52.8 | 51.3 | 32.3 | 55.3 | — | — | — | — | — | |
| DeepSeek-Coder-InstructSize=1.3B, Model Type=Instruction-Tuned Model, Setting=Zero-shot2024.01 | 48.4 | 65.2 | 45.3 | 51.9 | 45.3 | 59.7 | 55.1 | 12.7 | 52.2 | — | — | — | — | — | |
| UltraLink-LMBackbone=Llama-2-13b, SFT Data=UltraLink2024.02 | 46.8 | — | — | — | — | — | — | — | — | 60.4 | 43.9 | 40.9 | 49.4 | 39.6 | |
| CODE LLAMA - INSTRUCTSize=70B, decoding=greedy, mode=zero-shot2023.08 | 45.9 | — | 53.4 | 58.2 | 58.4 | 39 | 36.7 | 2,970 | — | — | — | — | — | — | |
| CODE LLAMASize=70B, decoding=greedy, mode=zero-shot2023.08 | 45.3 | — | 52.8 | 51.9 | 50.9 | 49.1 | 38 | 2,910 | — | — | — | — | — | — | |
| CODE LLAMA - PYTHONSize=70B, decoding=greedy, mode=zero-shot2023.08 | 45 | — | 54.7 | 57.6 | 53.4 | 44 | 34.8 | 2,530 | — | — | — | — | — | — | |
| DeepSeek-Coder-BaseSize=6.7B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 44.7 | 49.4 | 50.3 | 43 | 38.5 | 49.7 | 50 | 28.5 | 48.4 | — | — | — | — | — | |
| CodeLlamaSize=34B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 41 | 48.2 | 44.7 | 44.9 | 41 | 42.1 | 48.7 | 15.8 | 42.2 | — | — | — | — | — | |
| CODE LLAMASize=34B, decoding=greedy, mode=zero-shot2023.08 | 36.4 | — | 47.8 | 45.6 | 44.1 | 33.3 | 30.4 | 1,710 | — | — | — | — | — | — | |
| CODE LLAMA - INSTRUCTSize=34B, decoding=greedy, mode=zero-shot2023.08 | 36.1 | — | 45.3 | 43.7 | 36.6 | 40.3 | 31 | 1,960 | — | — | — | — | — | — | |
| CodeLlamaSize=13B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 35.4 | 36 | 37.9 | 38 | 34.2 | 45.2 | 43 | 16.5 | 32.3 | — | — | — | — | — | |
| CODE LLAMA - PYTHONSize=34B, decoding=greedy, mode=zero-shot2023.08 | 35.1 | — | 42.2 | 44.9 | 42.9 | 34.3 | 31.7 | 1,460 | — | — | — | — | — | — | |
| qa-FLoRABase LLM=LLaMA-3-8B, Paradigm=Data & Training free2025.12 | 34.4 | — | — | — | — | — | — | — | — | — | 48.2 | 31.1 | 23.8 | — | |
| LoRAFlowBase LLM=LLaMA-3-8B, Paradigm=Supervised2025.12 | 34.2 | — | — | — | — | — | — | — | — | — | 36.6 | 37.2 | 28.7 | — | |
| Centroid sim.Base LLM=LLaMA-3-8B, Paradigm=Training free2025.12 | 33.3 | — | — | — | — | — | — | — | — | — | 43.9 | 27.4 | 28.7 | — | |
| CODE LLAMA - INSTRUCTSize=13B, decoding=greedy, mode=zero-shot2023.08 | 32 | — | 42.2 | 40.5 | 32.3 | 39 | 24 | 1,390 | — | — | — | — | — | — | |
| CODE LLAMA - PYTHONSize=13B, decoding=greedy, mode=zero-shot2023.08 | 31.5 | — | 39.1 | 37.3 | 33.5 | 35.2 | 29.8 | 1,390 | — | — | — | — | — | — | |
| Avg [0.5, 0.5]Base LLM=LLaMA-3-8B, Paradigm=Static fusion2025.12 | 31.5 | — | — | — | — | — | — | — | — | — | 48.2 | 22.6 | 23.8 | — | |
| CODE LLAMASize=13B, decoding=greedy, mode=zero-shot2023.08 | 30.6 | — | 39.1 | 38 | 34.2 | 29.6 | 27.3 | 1,520 | — | — | — | — | — | — | |
| CodeLlamaSize=7B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 29.2 | 31.7 | 29.8 | 34.2 | 23.6 | 36.5 | 36.7 | 12 | 29.2 | — | — | — | — | — | |
| DeepSeek-Coder-BaseSize=1.3B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 28.3 | 34.8 | 31.1 | 32.3 | 24.2 | 28.9 | 36.7 | 10.1 | 28.6 | — | — | — | — | — | |
| StarCoderBaseSize=16B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 28 | 31.7 | 31.1 | 28.5 | 25.4 | 34 | 34.8 | 8.9 | 29.8 | — | — | — | — | — | |
| CODE LLAMA - PYTHONSize=7B, decoding=greedy, mode=zero-shot2023.08 | 27.5 | — | 32.3 | 35.4 | 32.4 | 23.9 | 24.7 | 1,650 | — | — | — | — | — | — | |
| CODE LLAMASize=7B, decoding=greedy, mode=zero-shot2023.08 | 26.3 | — | 28.6 | 34.2 | 24.2 | 33.3 | 25.3 | 1,200 | — | — | — | — | — | — | |
| code-cushman-001Size=12B, decoding=greedy, mode=zero-shot2023.08 | 26.1 | — | 30.6 | 31.9 | 28.9 | 31.3 | 22.1 | 1,170 | — | — | — | — | — | — | |
| CODE LLAMA - INSTRUCTSize=7B, decoding=greedy, mode=zero-shot2023.08 | 25.8 | — | 31.1 | 30.4 | 28.6 | 32.7 | 21.6 | 1,010 | — | — | — | — | — | — | |
| StarCoder PythonSize=15.5B, decoding=greedy, mode=zero-shot2023.08 | 25.3 | — | 31.6 | 30.2 | 26.1 | 32.3 | 21 | 1,050 | — | — | — | — | — | — | |
| StarCoder BaseSize=15.5B, decoding=greedy, mode=zero-shot2023.08 | 25 | — | 30.6 | 28.5 | 26.8 | 32.2 | 20.6 | 1,100 | — | — | — | — | — | — | |
| CodeGeeX2Size=6B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | 24.5 | 36 | 29.2 | 25.9 | 23.6 | 20.8 | 29.7 | 6.3 | 24.8 | — | — | — | — | — | |
| LLAMA-v2Size=70B, decoding=greedy, mode=zero-shot2023.08 | 24.4 | — | 30.4 | 31.7 | 34.2 | 15.1 | 25.9 | 890 | — | — | — | — | — | — | |
| LoRAFlowBase LLM=LLaMA-2-7B, Paradigm=Supervised2025.12 | 22.6 | — | — | — | — | — | — | — | — | — | 20.7 | 23.2 | 23.8 | — | |
| LoRAHubBase LLM=LLaMA-2-7B, Paradigm=Supervised2025.12 | 20.3 | — | — | — | — | — | — | — | — | — | 19.5 | 20.1 | 21.3 | — | |
| Centroid sim.Base LLM=LLaMA-2-7B, Paradigm=Training free2025.12 | 18.8 | — | — | — | — | — | — | — | — | — | 21.7 | 18.3 | 16.5 | — | |
| LLAMA-v2Size=34B, decoding=greedy, mode=zero-shot2023.08 | 18 | — | 23.6 | 22.2 | 19.9 | 21.4 | 17.1 | 380 | — | — | — | — | — | — | |
| Avg [0.5, 0.5]Base LLM=LLaMA-2-7B, Paradigm=Static fusion2025.12 | 17.7 | — | — | — | — | — | — | — | — | — | 17.1 | 18.3 | 17.7 | — | |
| qa-FLoRABase LLM=LLaMA-2-7B, Paradigm=Data & Training free2025.12 | 17.7 | — | — | — | — | — | — | — | — | — | 20.9 | 15.6 | 16.5 | — | |
| Chimera-inst-chat-13bBackbone=Llama-13b, SFT Data=Phoenix SFT data2024.02 | 13.9 | — | — | — | — | — | — | — | — | 14.6 | 13.4 | 14.6 | 12.8 | 14 | |
| CodeGen-MultiSize=16B, decoding=greedy, mode=zero-shot2023.08 | 13.4 | — | 21 | 22.2 | 8.4 | 20.1 | 8.2 | 60 | — | — | — | — | — | — | |
| Phoenix-inst-chat-7bBackbone=Bloomz-7b1, SFT Data=Phoenix SFT data2024.02 | 12.2 | — | — | — | — | — | — | — | — | 11 | 10.4 | 8.5 | 1.2 | 13.4 | |
| Guanaco-13bBackbone=Llama-2-13b, SFT Data=Guanaco Dataset2024.02 | 12.2 | — | — | — | — | — | — | — | — | 18.3 | 15.9 | 9.8 | 8.5 | 14.6 | |
| CodeGeeXSize=13B, decoding=greedy, mode=zero-shot2023.08 | 11.8 | — | 16.9 | 19.1 | 13.5 | 10.1 | 8.5 | 280 | — | — | — | — | — | — | |
| LLAMA-v2Size=13B, decoding=greedy, mode=zero-shot2023.08 | 11.4 | — | 13.7 | 15.8 | 13.1 | 13.2 | 9.5 | 320 | — | — | — | — | — | — | |
| Guanaco-7bBackbone=Llama-2-7b, SFT Data=Guanaco Dataset2024.02 | 9.9 | — | — | — | — | — | — | — | — | 9.2 | 6.7 | 11 | 9.8 | 12.8 | |
| Okapi-7bBackbone=Llama-2-7b, SFT Data=Okapi Dataset2024.02 | 9.8 | — | — | — | — | — | — | — | — | 12.2 | 11 | 8.5 | 8.5 | 8.5 | |
| LLAMA-v2Size=7B, decoding=greedy, mode=zero-shot2023.08 | 8.3 | — | 6.8 | 10.8 | 9.9 | 12.6 | 6.3 | 320 | — | — | — | — | — | — | |
| PolyLM-Chat-13bBackbone=PolyLM-13b, SFT Data=Closed-source2024.02 | 8.1 | — | — | — | — | — | — | — | — | 10.4 | 7.9 | 6.1 | 7.3 | 8.5 | |
| Bloomz-7b1-mtBackbone=Bloomz-7b1, SFT Data=xP3mt2024.02 | 7.3 | — | — | — | — | — | — | — | — | 8.5 | 7.3 | 6.1 | 8.5 | 6.1 | |
| PolyLM-Multialpaca-13bBackbone=PolyLM-13b, SFT Data=Multialpaca2024.02 | 6.8 | — | — | — | — | — | — | — | — | 8.5 | 7.3 | 6.1 | 6.1 | 6.1 | |
| code-cushman-001Size=12B, Model Type=Multilingual Base Model, Setting=Zero-shot2024.01 | — | 33.5 | 31.9 | 30.6 | 28.9 | 31.3 | 22.1 | 11.7 | — | — | — | — | — | — |