Code Generation on MBPP v1 (plus)
86.86Pass@1PaT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| PaTModel=Qwen332B2026.05 | 86.86 | 88.37 | 5.02 | 5.09 | |
| CodeTModel=Qwen332B2026.05 | 85.71 | 85.52 | 2.17 | 7.56 | |
| FunCoderModel=Qwen332B2026.05 | 85.71 | 87.66 | 4.31 | 8.93 | |
| PaTModel=DeepSeek-Coder2026.05 | 85.14 | 84.19 | 4.54 | 5.97 | |
| PaTModel=Qwen38B2026.05 | 84 | 85.58 | 7.94 | 5 | |
| PaTModel=Qwen314B2026.05 | 84 | 86.18 | 6.37 | 4.91 | |
| Best-of-NModel=Qwen332B2026.05 | 84 | 85.58 | 2.23 | 3.46 | |
| FunCoderModel=DeepSeek-Coder2026.05 | 84 | 83.6 | 3.95 | 8.77 | |
| FunCoderModel=Qwen314B2026.05 | 82.86 | 84.84 | 5.03 | 8.82 | |
| CodeTModel=DeepSeek-Coder2026.05 | 82.86 | 80.27 | 0.62 | 6.4 | |
| StandardModel=Qwen332B2026.05 | 82.29 | 83.35 | — | 1 | |
| FunCoderModel=Qwen38B2026.05 | 81.71 | 83.82 | 6.18 | 9.43 | |
| CodeTModel=Qwen314B2026.05 | 81.71 | 81.8 | 1.99 | 8.22 | |
| CodeTModel=Qwen38B2026.05 | 81.14 | 80 | 2.36 | 11.37 | |
| Best-of-NModel=Qwen314B2026.05 | 81.14 | 81.09 | 1.28 | 3.58 | |
| Best-of-NModel=DeepSeek-Coder2026.05 | 81.14 | 79.74 | 0.09 | 3.05 | |
| StandardModel=Qwen38B2026.05 | 79.43 | 77.64 | — | 1 | |
| StandardModel=DeepSeek-Coder2026.05 | 78.86 | 79.65 | — | 1 | |
| PaTModel=Qwen34B2026.05 | 78.29 | 83.13 | 7.08 | 4.85 | |
| StandardModel=Qwen314B2026.05 | 78.29 | 79.81 | — | 1 | |
| FunCoderModel=Qwen34B2026.05 | 77.71 | 81.18 | 5.13 | 8.31 | |
| Best-of-NModel=Qwen38B2026.05 | 77.71 | 77.83 | 0.19 | 3.5 | |
| CodeTModel=Qwen34B2026.05 | 77.14 | 76.63 | 0.58 | 18.82 | |
| Best-of-NModel=Qwen34B2026.05 | 76.57 | 76.95 | 0.9 | 3.39 | |
| StandardModel=Qwen34B2026.05 | 76 | 76.05 | — | 1 | |
| PaTModel=Llama3.18B2026.05 | 72 | 73.31 | 8.16 | 5.32 | |
| CodeTModel=Llama3.18B2026.05 | 71.43 | 69.24 | 4.09 | 4.86 | |
| FunCoderModel=Llama3.18B2026.05 | 71.43 | 71.53 | 6.38 | 8.07 | |
| Best-of-NModel=Llama3.18B2026.05 | 69.14 | 68.51 | 3.36 | 2.6 | |
| StandardModel=Llama3.18B2026.05 | 66.29 | 65.15 | — | 1 |