Code Generation on HumanEval-X
67.1Pass@1 (C++)INTERVENOR
Evaluation results
| Method | Links | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| INTERVENOR2023.11 | 67.1 | — | — | — | 68.3 | 67.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MagicoderS-DSBase Model=DeepSeek-Coder-Base, Parameters=6.7B, Protocol=Zero-shot2024.05 | 63.4 | — | — | — | 70.7 | 66.5 | 72.6 | — | — | — | — | — | — | — | — | 51.8 | — | — | 65 | — | — | — | — | — | |
| AlchemistCoder-DSBase Model=DeepSeek-Coder-Base, Parameters=6.7B, Protocol=Zero-shot2024.05 | 62.2 | — | — | — | 72 | 68.9 | 79.9 | — | — | — | — | — | — | — | — | 59.8 | — | — | 68.6 | — | — | — | — | — | |
| AlchemistCoder-CLBase Model=CodeLlama-Python, Parameters=7B, Protocol=Zero-shot2024.05 | 53.1 | — | — | — | 64 | 52.4 | 74.4 | — | — | — | — | — | — | — | — | 42.7 | — | — | 57.3 | — | — | — | — | — | |
| GPT-3.52023.11 | 52.4 | — | — | — | 50.6 | 54.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MagicoderS-CLBase Model=CodeLlama-Python, Parameters=7B, Protocol=Zero-shot2024.05 | 47.6 | — | — | — | 34.8 | 57.9 | 68.3 | — | — | — | — | — | — | — | — | 39.6 | — | — | 49.6 | — | — | — | — | — | |
| DeepSeek-Coder-BaseParameters=6.7B, Protocol=Zero-shot2024.05 | 45.1 | — | — | — | 56.1 | 43.9 | 47.6 | — | — | — | — | — | — | — | — | 38.4 | — | — | 46.2 | — | — | — | — | — | |
| CodeLlama-PythonParameters=7B, Protocol=Zero-shot2024.05 | 33.5 | — | — | — | 39.6 | 35.4 | 37.8 | — | — | — | — | — | — | — | — | 30.5 | — | — | 35.4 | — | — | — | — | — | |
| AlchemistCoder-LBase Model=Llama 2, Parameters=7B, Protocol=Zero-shot2024.05 | 31.1 | — | — | — | 45.1 | 41.5 | 56.7 | — | — | — | — | — | — | — | — | 25.6 | — | — | 37.1 | — | — | — | — | — | |
| CodeLlamaParameters=7B, Protocol=Zero-shot2024.05 | 27.4 | — | — | — | 25.6 | 32.9 | 31.7 | — | — | — | — | — | — | — | — | 12.8 | — | — | 26.1 | — | — | — | — | — | |
| CodeGen2023.11 | 18.1 | — | — | — | 14.9 | 18.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeGen-Multi-16BParameters=16B, Type=Multi2023.03 | 18.05 | — | — | — | 14.95 | 18.4 | 19.22 | 34.64 | 55.17 | 30.84 | 50.9 | 36.73 | 60.62 | 32.8 | 56.48 | 13.03 | 25.46 | 48.77 | 16.73 | 32.09 | 54.39 | — | — | — | |
| CodeGeeX2023.11 | 17.1 | — | — | — | 20 | 17.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeGeeX-13BParameters=13B2023.03 | 17.06 | — | — | — | 20.04 | 17.59 | 22.89 | 39.57 | 60.92 | 32.21 | 51 | 36.7 | 58.42 | 32.28 | 56.33 | 14.43 | 25.68 | 47.14 | 18.4 | 33.29 | 54.76 | — | — | — | |
| CodeGen-Multi-6BParameters=6B, Type=Multi2023.03 | 11.44 | — | — | — | 15.17 | 15.41 | 19.41 | 30.29 | 49.63 | 26.23 | 42.82 | 31.74 | 53.91 | 27.92 | 48.81 | 9.98 | 23.26 | 41.01 | 14.28 | 27.89 | 47.24 | — | — | — | |
| Llama 2Parameters=7B, Protocol=Zero-shot2024.05 | 11 | — | — | — | 11 | 14 | 14 | — | — | — | — | — | — | — | — | 6.1 | — | — | 11.2 | — | — | — | — | — | |
| GPT-NeoX-20BParameters=20B2023.03 | 9.9 | — | — | — | 8.87 | 11.28 | 13.83 | 22.72 | 39.56 | 18.99 | 38.75 | 19.55 | 42.23 | 20.78 | 42.67 | 5 | 15.7 | 32.08 | 9.78 | 19.55 | 39.06 | — | — | — | |
| InCoder2023.11 | 9.5 | — | — | — | 9.1 | 13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InCoder-6.7BParameters=6.7B2023.03 | 9.5 | — | — | — | 9.05 | 12.98 | 16.41 | 26.55 | 43.95 | 19.3 | 36.1 | 18.64 | 40.7 | 22.98 | 43.34 | 8.68 | 13.8 | 28.31 | 11.33 | 20.25 | 38.48 | — | — | — | |
| GPT-J-6BParameters=6B2023.03 | 7.54 | — | — | — | 7.86 | 8.99 | 11.1 | 18.67 | 30.98 | 13.67 | 30.16 | 14.37 | 32.96 | 16.32 | 33.77 | 4.01 | 10.81 | 23.7 | 7.9 | 14.77 | 30.32 | — | — | — | |
| ChatGLM-3-6BSampling strategy=Greedy, Note=Baseline2024.02 | — | 18.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Falcon-40BSampling strategy=Greedy, Note=Baseline2024.02 | — | 20.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InternLM-20BSampling strategy=Greedy, Note=Baseline2024.02 | — | 21.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-2-13BSampling strategy=Greedy, Note=Baseline2024.02 | — | 20.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-2-70BSampling strategy=Greedy, Note=Baseline2024.02 | — | 28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-2-7BSampling strategy=Greedy, Note=Baseline2024.02 | — | 18.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen-14BSampling strategy=Greedy, Note=Baseline2024.02 | — | 26.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-4B + FBS-Full (ours)Backbone=Qwen3-4B2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 46.2 | — | — | 532 | 0.7 | 36 | |
| SPACEBackbone=ChatGLM-3-6B, Sampling strategy=Greedy2024.02 | — | 18.3 | 3.34 | 2.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPACEBackbone=LLaMA-2-7B, Sampling strategy=Greedy2024.02 | — | 18.9 | 3.54 | 3.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPACEBackbone=LLaMA-2-13B, Sampling strategy=Greedy2024.02 | — | 20.1 | 3.76 | 3.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPACEBackbone=Qwen-14B, Sampling strategy=Greedy2024.02 | — | 26.8 | 3.51 | 3.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPACEBackbone=InternLM-20B, Sampling strategy=Greedy2024.02 | — | 21.3 | 3.31 | 3.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPACEBackbone=Falcon-40B, Sampling strategy=Greedy2024.02 | — | 20.7 | 3.58 | 3.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPACEBackbone=LLaMA-2-70B, Sampling strategy=Greedy2024.02 | — | 28 | 4.32 | 4.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |