Code Generation on EvoEval semantic-altering
48Pass@1 AverageCodeQwen1.5-7B-Chat
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| CodeQwen1.5-7B-ChatParameters=7B, Type=Chat2024.10 | 48 | 39 | 38 | 71 | 31 | 61 | |
| SelfCodeAlign-CQ-7BParameters=7B2024.10 | 43.6 | 33 | 40 | 60 | 20 | 65 | |
| CodeQwen1.5-7B-OctoPackParameters=7B, Type=OctoPack2024.10 | 42.2 | 35 | 36 | 59 | 22 | 59 | |
| DeepSeek-Coder-6.7B-InstructParameters=6.7B, Type=Instruct2024.10 | 41.4 | 40 | 37 | 61 | 18 | 51 | |
| Llama-3-8B-InstructParameters=8B, Type=Instruct2024.10 | 40.6 | 34 | 39 | 57 | 15 | 58 | |
| CodeQwen1.5-7B-BaseParameters=7B, Type=Base2024.10 | 36.2 | 26 | 30 | 46 | 18 | 61 | |
| CodeGemma-7B-ITParameters=7B, Type=IT2024.10 | 35.4 | 31 | 32 | 49 | 9 | 56 | |
| OctoCoder-16BParameters=16B2024.10 | 30.6 | 19 | 26 | 43 | 11 | 54 | |
| StarCoder2-15BParameters=15B2024.10 | 25.8 | 16 | 19 | 41 | 5 | 48 |