Safe Code Generation on Java Safe Code Generation Dataset
97.18Safe Code RateClaude Sonnet 4.6
Evaluation Results
| Method | Links | |
|---|---|---|
| Claude Sonnet 4.6Prompting Condition=+Comp. I (SPARK Component I)2026.06 | 97.18 | |
| Claude Sonnet 4.6Prompting Condition=Baseline2026.06 | 94.37 | |
| GPT-5.4Prompting Condition=+Comp. I (SPARK Component I)2026.06 | 93.66 | |
| Claude Haiku 4.5Prompting Condition=+Comp. I (SPARK Component I)2026.06 | 90.85 | |
| GPT-5.5Prompting Condition=+Comp. I (SPARK Component I)2026.06 | 89.84 | |
| Claude Haiku 4.5Prompting Condition=Baseline2026.06 | 88.73 | |
| GPT-5.5Prompting Condition=Baseline2026.06 | 85.71 | |
| DeepSeek V4 ProPrompting Condition=+Comp. I (SPARK Component I)2026.06 | 83.1 | |
| GPT-5.4-miniPrompting Condition=+Comp. I (SPARK Component I)2026.06 | 83.1 | |
| GPT-5.4Prompting Condition=Baseline2026.06 | 79.58 | |
| DeepSeek V4 FlashPrompting Condition=+Comp. I (SPARK Component I)2026.06 | 76.76 | |
| GPT-5.4-miniPrompting Condition=Baseline2026.06 | 69.72 | |
| DeepSeek V4 ProPrompting Condition=Baseline2026.06 | 58.45 | |
| DeepSeek V4 FlashPrompting Condition=Baseline2026.06 | 48.59 | |
| Claude Haiku 4.5Prompting Condition=Plain2026.06 | 36.62 | |
| GPT-5.4Prompting Condition=Plain2026.06 | 35.92 | |
| GPT-5.5Prompting Condition=Plain2026.06 | 31.69 | |
| Claude Sonnet 4.6Prompting Condition=Plain2026.06 | 30.99 | |
| GPT-5.4-miniPrompting Condition=Plain2026.06 | 28.17 | |
| DeepSeek V4 ProPrompting Condition=Plain2026.06 | 25.35 | |
| DeepSeek V4 FlashPrompting Condition=Plain2026.06 | 21.13 |