Code Generation on MBPP Sanitized (test)
84.3Pass@1GPT-4o
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-4oPrompting Method=Personality2024.10 | 84.3 | 91.6 | 93.9 | |
| GPT-4o miniPrompting Method=Personality2024.10 | 82.2 | 86.4 | 92 | |
| Llama 3.1Prompting Method=Personality2024.10 | 81 | 85.5 | 92.3 | |
| GPT-4oPrompting Method=Direct2024.10 | 78.2 | 80.1 | 82 | |
| CodestralPrompting Method=Personality2024.10 | 73.8 | 85 | 92.7 | |
| Llama 3.1Prompting Method=Direct2024.10 | 69.8 | 74 | 76.6 | |
| GPT-4o miniPrompting Method=Direct2024.10 | 69.3 | 74.7 | 77.8 | |
| CodestralPrompting Method=Direct2024.10 | 64.2 | 73.8 | 79.2 | |
| CodeLlamaPrompting Method=Personality2024.10 | 46.8 | 49.6 | 51.2 | |
| CodeLlamaPrompting Method=Direct2024.10 | 43.3 | 46.6 | 47.1 |