Rule-level Identification on SecGenEval-PS CodeAnalysis
85.2Success Rate @1 Ruleo3-mini
Evaluation results
| Method | Links | ||
|---|---|---|---|
| o3-miniEvaluation Mode=M22026.01 | 85.2 | 60.8 | |
| o3-miniEvaluation Mode=M32026.01 | 84.5 | 68.6 | |
| GPT-4oEvaluation Mode=M32026.01 | 80.7 | 47.5 | |
| GPT-4oEvaluation Mode=M22026.01 | 78 | 40.8 | |
| GPT-4oEvaluation Mode=M12026.01 | 55.2 | 26.7 | |
| Qwen2.5-7BEvaluation Mode=M12026.01 | 50 | 0 | |
| o3-miniEvaluation Mode=M12026.01 | 13.6 | 10.8 | |
| DeepSeek-R1-Distill-Qwen-7BEvaluation Mode=M12026.01 | 13.2 | 4 | |
| Qwen2.5-Coder-7BEvaluation Mode=M32026.01 | 8.3 | 2.1 | |
| Qwen2.5-Coder-7BEvaluation Mode=M22026.01 | 1.5 | 0 | |
| Qwen2.5-Coder-7BEvaluation Mode=M12026.01 | 0 | 0 |