Code Generation on MBPP sample 200
79.5Pass@1FUNCODER
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FUNCODERModel=GPT-42024.05 | 79.5 | 6 | |
| FUNCODERModel=GPT-3.52024.05 | 78.5 | 6.5 | |
| ReflexionModel=GPT-42024.05 | 77.1 | 3.6 | |
| CodeTModel=GPT-42024.05 | 77 | 3.5 | |
| CodeTModel=GPT-3.52024.05 | 76 | 4 | |
| LDBModel=GPT-3.52024.05 | 76 | 4 | |
| StandardModel=GPT-42024.05 | 73.5 | — | |
| ReflexionModel=GPT-3.52024.05 | 72.5 | 0.5 | |
| StandardModel=GPT-3.52024.05 | 72 | — | |
| FUNCODERModel=StableCode-3b2024.05 | 63.5 | 12 | |
| FUNCODERModel=Llama3-8b2024.05 | 62.5 | 2 | |
| CodeTModel=Llama3-8b2024.05 | 61.5 | 1 | |
| StandardModel=Llama3-8b2024.05 | 60.5 | — | |
| FUNCODERModel=CodeLlama-34b2024.05 | 58.5 | 5 | |
| CodeTModel=StableCode-3b2024.05 | 57.5 | 6 | |
| CodeTModel=CodeLlama-34b2024.05 | 56.5 | 3 | |
| StandardModel=CodeLlama-34b2024.05 | 53.5 | — | |
| StandardModel=StableCode-3b2024.05 | 51.5 | — |