Secure Code Generation on CWEval (pass@X metrics)
48.2pass@1Vanilla
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| VanillaModel=AR2026.05 | 48.2 | — | 24.1 | — | 21.3 | 36.1 | 40.7 | |
| RepromptModel=AR2026.05 | 44.4 | — | 47.2 | — | 39.8 | 63 | 68.5 | |
| Vanilla ModelUnderlying Model=Deepseek-Coder-V2-Lite2026.03 | 42.17 | 70.03 | 16.83 | 37.02 | — | — | — | |
| CodeGuard+Underlying Model=Deepseek-Coder-V2-Lite2026.03 | 42.04 | 68.05 | 19.18 | 41.63 | — | — | — | |
| SCS-CodeUnderlying Model=Deepseek-Coder-V2-Lite2026.03 | 41.23 | 67.59 | 20.09 | 42.69 | — | — | — | |
| Vanilla ModelUnderlying Model=CodeLlama-7b2026.03 | 40.46 | 70.17 | 19.03 | 38.89 | — | — | — | |
| Vanilla ModelUnderlying Model=Mistral-7b2026.03 | 40.37 | 68.52 | 17.02 | 35.81 | — | — | — | |
| Hybrid (CodeGuard + SCS)Underlying Model=Deepseek-Coder-V2-Lite2026.03 | 40.28 | 66.87 | 19.72 | 40.82 | — | — | — | |
| SCS-CodeUnderlying Model=Mistral-7b2026.03 | 39.91 | 68.52 | 18.9 | 38.96 | — | — | — | |
| CDCModel=Diffusion2026.05 | 39.8 | — | 41.7 | — | 34.3 | 49.1 | 51.9 | |
| SCS-CodeUnderlying Model=CodeLlama-7b2026.03 | 38.89 | 69.52 | 20.11 | 41.68 | — | — | — | |
| Hybrid (CodeGuard + SCS)Underlying Model=Llama3.1-8b2026.03 | 37.64 | 70.37 | 17.37 | 5.19 | — | — | — | |
| SCS-CodeUnderlying Model=Llama3.1-8b2026.03 | 37.22 | 69.44 | 16.81 | 38.28 | — | — | — | |
| Hybrid (Prefix + SCS)Underlying Model=Deepseek-Coder-V2-Lite2026.03 | 35.33 | 69.44 | 19.66 | 44.44 | — | — | — | |
| CodeGuard+Underlying Model=Llama3.1-8b2026.03 | 35.08 | 66.48 | 15.87 | 34.19 | — | — | — | |
| Vanilla ModelUnderlying Model=Llama3.1-8b2026.03 | 33.34 | 64.87 | 13.58 | 30.57 | — | — | — | |
| Secure Code PrefixUnderlying Model=Deepseek-Coder-V2-Lite2026.03 | 32.99 | 65.74 | 17.04 | 40.74 | — | — | — | |
| Hybrid (Prefix + SCS)Underlying Model=Llama3.1-8b2026.03 | 30.17 | 69.44 | 13.89 | 37.96 | — | — | — | |
| Secure Code PrefixUnderlying Model=Llama3.1-8b2026.03 | 27.98 | 66.6 | 12.96 | 36.11 | — | — | — | |
| Secure Code PrefixUnderlying Model=CodeLlama-7b2026.03 | 27.43 | 70.31 | 13.08 | 51.04 | — | — | — | |
| VanillaModel=Diffusion2026.05 | 26.9 | — | 18.5 | — | 12 | 25.9 | 26.9 | |
| Sec promptModel=Diffusion2026.05 | 26.9 | — | 19.4 | — | 13.9 | 23.2 | 27.8 | |
| Hybrid (SafeCoder + SCS)Underlying Model=Mistral-7b2026.03 | 26.74 | 49.07 | 13.67 | 31.56 | — | — | — | |
| Hybrid (Prefix + SCS)Underlying Model=CodeLlama-7b2026.03 | 26.55 | 71.3 | 15.22 | 55.56 | — | — | — | |
| SafeCoderUnderlying Model=Mistral-7b2026.03 | 25.49 | 53.7 | 12.07 | 35.19 | — | — | — | |
| Hybrid (Prefix + SCS)Underlying Model=Mistral-7b2026.03 | 22.98 | 62.96 | 12.82 | 47.42 | — | — | — | |
| Secure Code PrefixUnderlying Model=Mistral-7b2026.03 | 22.05 | 62.9 | 11.84 | 46.3 | — | — | — | |
| Hybrid (SafeCoder + SCS)Underlying Model=CodeLlama-7b2026.03 | 9.55 | 29.83 | 5.4 | 21.76 | — | — | — | |
| SafeCoderUnderlying Model=CodeLlama-7b2026.03 | 8.8 | 27.81 | 4.55 | 19.45 | — | — | — |