Code Generation on MultiPL-E HumanEval translated from Python
54.6C++ Pass RateIF-CL
Evaluation results
| Method | Links | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IF-CLSize=13B2023.12 | 54.6 | — | — | — | 45.3 | — | 64.6 | — | 53.4 | — | — | — | — | — | 54.5 | — | — | 50 | — | — | |
| WizardCoder-CLPSize=34B2023.12 | 47.2 | — | — | — | 44.9 | — | 55.3 | — | 47.2 | — | — | — | — | — | 46.2 | — | — | 44.3 | — | — | |
| IF-CL (-fused inst.)Size=13B2023.12 | 47.2 | — | — | — | 39.8 | — | 55.3 | — | 44.7 | — | — | — | — | — | 44.9 | — | — | 41.8 | — | — | |
| MagicoderS-CLPSize=7B2023.12 | 44.4 | — | — | — | 42.9 | — | 57.5 | — | 47.6 | — | — | — | — | — | 40.3 | — | — | 44.1 | — | — | |
| CODELLAMA-INSTRUCTSize=34B2023.12 | 41.5 | — | — | — | 41.5 | — | 45.9 | — | 37 | — | — | — | — | — | 39.3 | — | — | 37.6 | — | — | |
| CODELLAMASize=34B2023.12 | 41.4 | — | — | — | 40.2 | — | 41.7 | — | 40.4 | — | — | — | — | — | 38.7 | — | — | 35.3 | — | — | |
| CODELLAMA-PYTHONSize=34B2023.12 | 39.1 | — | — | — | 39.5 | — | 44.7 | — | 39.8 | — | — | — | — | — | 39.7 | — | — | 34.3 | — | — | |
| WizardCoder-SCSize=15B2023.12 | 39 | — | — | — | 35.8 | — | 41.9 | — | 39.3 | — | — | — | — | — | 27.1 | — | — | 33.7 | — | — | |
| StarCoderSize=15B2023.12 | 31.6 | — | — | — | 30.2 | — | 30.8 | — | 26.1 | — | — | — | — | — | 21.8 | — | — | 22.7 | — | — | |
| StarCoderAccess=open-access2023.05 | 31.55 | 21.01 | 13.57 | 17.61 | 30.22 | 23.02 | 30.79 | 23.89 | 26.08 | 17.34 | 33.57 | 15.5 | 1.24 | 0.07 | 21.84 | 27.61 | 10.46 | 22.74 | 32.29 | — | |
| StarCoderBaseSize=15B2023.12 | 30.6 | — | — | — | 28.5 | — | 31.7 | — | 26.8 | — | — | — | — | — | 24.5 | — | — | 16.7 | — | — | |
| code-cushman-001Access=closed-access2023.05 | 30.59 | 22.06 | 6.73 | 19.68 | 31.9 | 1.54 | 31.27 | 26.24 | 28.94 | 19.29 | 30.71 | 10.99 | 28.63 | 7.05 | 25.22 | 27.62 | 11.74 | 22.12 | 31.26 | — | |
| StarCoderBaseAccess=open-access2023.05 | 30.56 | 20.56 | 10.01 | 21.47 | 28.53 | 21.09 | 31.7 | 26.61 | 26.75 | 16.32 | 30.35 | 10.18 | 17.25 | 11.77 | 24.46 | 28.79 | 11.02 | 16.74 | 32.15 | — | |
| CODELLAMA-PYTHONSize=7B2023.12 | 30.2 | — | — | — | 29.1 | — | 35.7 | — | 29 | — | — | — | — | — | 27 | — | — | 27.1 | — | — | |
| CODELLAMASize=7B2023.12 | 27 | — | — | — | 29.3 | — | 31.7 | — | 25.1 | — | — | — | — | — | 25.5 | — | — | 25.6 | — | — | |
| CodeGen-16B-MultiAccess=open-access, Parameters=16B2023.05 | 21 | 8.24 | 7.68 | 13.54 | 22.2 | 0 | 19.15 | 8.5 | 8.37 | 3.42 | 19.26 | 6.45 | 0 | 0.66 | 4.21 | 2.37 | 0.61 | 1.25 | 20.07 | — | |
| CodeGeeXAccess=open-access2023.05 | 16.87 | 8.49 | 9.15 | 11.04 | 19.14 | 0.29 | 16.92 | 10.96 | 13.51 | 8.09 | 21.62 | 3.92 | 3.34 | 3.31 | 7.88 | 8.95 | 2.75 | 7.26 | 10.11 | — | |
| DeepSeek V3.1Model Variant=Base, # Shots=0-shot, # Activated Params=37B, # Total Params=671B2026.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 45.9 | |
| DeepSeek V3.2Model Variant=Exp Base, # Shots=0-shot, # Activated Params=37B, # Total Params=671B2026.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 45.7 | |
| DeepSeek-V3.1 Base# Shots=0-shot, # Activated Params=37B, # Total Params=671B2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 45.9 | |
| DeepSeek-V3.2 Exp Base# Shots=0-shot, # Activated Params=37B, # Total Params=671B2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 45.7 | |
| Kimi-K2Model Variant=Base, # Shots=0-shot, # Activated Params=32B, # Total Params=1043B2026.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60.5 | |
| Kimi-K2 Base# Shots=0-shot, # Activated Params=32B, # Total Params=1043B2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60.5 | |
| MiMo-V2 FlashModel Variant=Base, # Shots=0-shot, # Activated Params=15B, # Total Params=309B2026.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.5 | |
| MiMo-V2-Flash Base# Shots=0-shot, # Activated Params=15B, # Total Params=309B2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.5 | |
| Step 3.5 FlashModel Variant=Base, # Shots=0-shot, # Activated Params=11B, # Total Params=196B2026.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.7 |