Issue-level Localization on SecGenEval-PS CodeAnalysis
59.1Success Rate @ Issueo3-mini
Evaluation results
| Method | Links | ||
|---|---|---|---|
| o3-miniEvaluation Mode=M22026.01 | 59.1 | 34.7 | |
| o3-miniEvaluation Mode=M32026.01 | 56.7 | 37.9 | |
| GPT-4oEvaluation Mode=M32026.01 | 25.1 | 6.7 | |
| GPT-4oEvaluation Mode=M22026.01 | 23.3 | 6 | |
| GPT-4oEvaluation Mode=M12026.01 | 13.6 | 4 | |
| o3-miniEvaluation Mode=M12026.01 | 6.8 | 2.4 | |
| DeepSeek-R1-Distill-Qwen-7BEvaluation Mode=M12026.01 | 1.3 | 0.9 | |
| Qwen2.5-Coder-7BEvaluation Mode=M12026.01 | 0 | 0 | |
| Qwen2.5-Coder-7BEvaluation Mode=M22026.01 | 0 | 0 | |
| Qwen2.5-Coder-7BEvaluation Mode=M32026.01 | 0 | 0 | |
| Qwen2.5-7BEvaluation Mode=M12026.01 | 0 | 0 |