Code Generation on HumanEval base v1 (Pass@1, Cost)
93.9Pass@1PaT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| PaTModel=Qwen332B2026.05 | 93.9 | 88.37 | 5.02 | 5.09 | |
| FunCoderModel=Qwen332B2026.05 | 93.29 | 87.66 | 4.31 | 8.93 | |
| PaTModel=Qwen314B2026.05 | 91.46 | 86.18 | 6.37 | 4.91 | |
| PaTModel=Qwen38B2026.05 | 90.85 | 85.58 | 7.94 | 5 | |
| Best-of-NModel=Qwen332B2026.05 | 90.24 | 85.58 | 2.23 | 3.46 | |
| PaTModel=Qwen34B2026.05 | 89.63 | 83.13 | 7.08 | 4.85 | |
| FunCoderModel=Qwen314B2026.05 | 89.63 | 84.84 | 5.03 | 8.82 | |
| FunCoderModel=Qwen38B2026.05 | 89.02 | 83.82 | 6.18 | 9.43 | |
| CodeTModel=Qwen332B2026.05 | 89.02 | 85.52 | 2.17 | 7.56 | |
| StandardModel=Qwen332B2026.05 | 87.2 | 83.35 | — | 1 | |
| PaTModel=DeepSeek-Coder2026.05 | 86.59 | 84.19 | 4.54 | 5.97 | |
| FunCoderModel=Qwen34B2026.05 | 85.98 | 81.18 | 5.13 | 8.31 | |
| FunCoderModel=DeepSeek-Coder2026.05 | 85.98 | 83.6 | 3.95 | 8.77 | |
| StandardModel=Qwen314B2026.05 | 83.54 | 79.81 | — | 1 | |
| CodeTModel=Qwen314B2026.05 | 83.54 | 81.8 | 1.99 | 8.22 | |
| StandardModel=DeepSeek-Coder2026.05 | 83.54 | 79.65 | — | 1 | |
| Best-of-NModel=Qwen314B2026.05 | 82.93 | 81.09 | 1.28 | 3.58 | |
| Best-of-NModel=DeepSeek-Coder2026.05 | 81.1 | 79.74 | 0.09 | 3.05 | |
| CodeTModel=DeepSeek-Coder2026.05 | 81.1 | 80.27 | 0.62 | 6.4 | |
| Best-of-NModel=Qwen38B2026.05 | 80.49 | 77.83 | 0.19 | 3.5 | |
| CodeTModel=Qwen38B2026.05 | 80.49 | 80 | 2.36 | 11.37 | |
| Best-of-NModel=Qwen34B2026.05 | 79.27 | 76.95 | 0.9 | 3.39 | |
| StandardModel=Qwen34B2026.05 | 78.66 | 76.05 | — | 1 | |
| CodeTModel=Qwen34B2026.05 | 78.66 | 76.63 | 0.58 | 18.82 | |
| StandardModel=Qwen38B2026.05 | 78.66 | 77.64 | — | 1 | |
| PaTModel=Llama3.18B2026.05 | 78.05 | 73.31 | 8.16 | 5.32 | |
| FunCoderModel=Llama3.18B2026.05 | 75 | 71.53 | 6.38 | 8.07 | |
| CodeTModel=Llama3.18B2026.05 | 71.95 | 69.24 | 4.09 | 4.86 | |
| Best-of-NModel=Llama3.18B2026.05 | 70.12 | 68.51 | 3.36 | 2.6 | |
| StandardModel=Llama3.18B2026.05 | 68.29 | 65.15 | — | 1 |