Code Generation on HumanEval compile (L0)
26.95Pass@1ShieldedCode
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ShieldedCode2026.01 | 26.95 | 35.68 | |
| GPT-4o2026.01 | 22.58 | 31.47 | |
| DeepSeekCoder-7BParameters=7B2026.01 | 10.28 | 14.23 | |
| CodeLlama2026.01 | 7.84 | 9.21 | |
| GPT-3.5-Turbo2026.01 | 6.89 | 10.18 | |
| Meta LLMCompiler-7BParameters=7B2026.01 | 6.42 | 7.64 | |
| StarCoder2-7BParameters=7B2026.01 | 5.78 | 9.45 | |
| Qwen-2.5-Coder-7BParameters=7B2026.01 | 5.31 | 7.12 |