Secure Code Generation on CVS (test)
98C++ Success RateLlama3-70b-instruct
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Llama3-70b-instructIntegrated with INDICT=true2024.06 | 98 | 94 | 90 | 98 | 94 | 94.8 | |
| Llama3-8b-instructIntegrated with INDICT=true2024.06 | 91 | 95 | 90 | 97 | 90 | 92.6 | |
| GT Secure Code2024.06 | 83 | 93 | 86 | 90 | 92 | 88.8 | |
| CommandRIntegrated with INDICT=true2024.06 | 75 | 100 | 90 | 96 | 91 | 90.4 | |
| Llama3-70b-instructIntegrated with INDICT=false2024.06 | 63 | 83 | 97 | 88 | 95 | 85.2 | |
| Llama3-8b-instructIntegrated with INDICT=false2024.06 | 43 | 84 | 91 | 93 | 98 | 81.8 | |
| GT Unsecure Code2024.06 | 35 | 63 | 84 | 88 | 93 | 72.6 | |
| CommandRIntegrated with INDICT=false2024.06 | 31 | 90 | 87 | 85 | 93 | 77.2 |