Code Generation on MBPP+ (Pass rate, Change)
72.7Pass RateGPT-4o
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPT-4oStrategy=MBTI2024.10 | 72.7 | 1.5 | |
| DeepSeek-Coder V2Strategy=MBTI2024.10 | 72.2 | 0.8 | |
| GPT-4o miniStrategy=MBTI2024.10 | 71.7 | 2.3 | |
| DeepSeek-Coder V2Strategy=Direct2024.10 | 71.4 | — | |
| GPT-4oStrategy=Direct2024.10 | 71.2 | — | |
| Qwen-LongStrategy=MBTI2024.10 | 71.2 | 3.5 | |
| GPT-4o miniStrategy=Direct2024.10 | 69.4 | — | |
| Llama3.1Strategy=MBTI2024.10 | 69.2 | 2.5 | |
| Qwen-LongStrategy=Direct2024.10 | 67.7 | — | |
| Llama3.1Strategy=Direct2024.10 | 66.7 | — | |
| CodestralStrategy=MBTI2024.10 | 64.9 | 3.7 | |
| CodestralStrategy=Direct2024.10 | 61.2 | — | |
| VanillaStep=2502026.02 | 54.2 | — | |
| VanillaStep=3002026.02 | 54 | — | |
| CFStep=3002026.02 | 53.7 | -0.3 | |
| VanillaStep=2002026.02 | 53.4 | — | |
| CFStep=252026.02 | 52.7 | 1.1 | |
| CodeLlamaStrategy=MBTI2024.10 | 52.4 | 10 | |
| CFStep=1002026.02 | 52.4 | 0.5 | |
| CFStep=2002026.02 | 52.1 | -1.3 | |
| VanillaStep=1002026.02 | 51.9 | — | |
| VanillaStep=252026.02 | 51.6 | — | |
| CFStep=2502026.02 | 51.3 | -2.9 | |
| CodeLlamaStrategy=Direct2024.10 | 42.4 | — |