Secure Code Generation on COBALT Security Prompts 500 prompts per model
48.4Vulnerability RateGemini 2.5 Flash
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Gemini 2.5 FlashTemperature=0, API Identifier=gemini-2.5-flash2026.04 | 48.4 | 146 | 86 | 142 | — | |
| Claude Haiku 4.5Temperature=0, API Identifier=claude-haiku-4-5-202510012026.04 | 49.2 | 155 | 81 | 152 | — | |
| GPT-4.1Temperature=0, API Identifier=gpt-4.12026.04 | 54 | 142 | 86 | 136 | — | |
| Mistral LargeTemperature=0, API Identifier=mistral-large-latest2026.04 | 57.8 | 155 | 94 | 155 | — | |
| Llama 3.3 70BTemperature=0, API Identifier=llama-3.3-70b-versatile2026.04 | 58.4 | 168 | 83 | 147 | — | |
| Llama 4 ScoutTemperature=0, API Identifier=llama-4-scout-17b-16e-instruct2026.04 | 60.6 | 167 | 95 | 156 | — | |
| GPT-4oTemperature=0, API Identifier=gpt-4o2026.04 | 62.4 | 166 | 106 | 167 | — |