Code Generation on HumanEval compile (L2)
19.23Pass@1ShieldedCode
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ShieldedCode2026.01 | 19.23 | 29.71 | |
| GPT-4o2026.01 | 15.26 | 22.36 | |
| DeepSeekCoder-7BParameters=7B2026.01 | 7.94 | 12.18 | |
| StarCoder2-7BParameters=7B2026.01 | 6.23 | 6.82 | |
| Qwen-2.5-Coder-7BParameters=7B2026.01 | 6.08 | 5.94 | |
| Meta LLMCompiler-7BParameters=7B2026.01 | 5.97 | 7.28 | |
| CodeLlama2026.01 | 5.19 | 7.89 | |
| GPT-3.5-Turbo2026.01 | 4.87 | 6.74 |