Vulnerability Detection on Vulnerability Detection Benchmark
60.27Pass@1DeepSeek-V3.1
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V3.1Training Paradigm=Zero-Shot Prompting2026.02 | 60.27 | 71.27 | 26.09 | 82.47 | 39.64 | 22.5 | 71.95 | 3.59 | 1.95 | |
| GPT-OSS-120BTraining Paradigm=Zero-Shot Prompting, Model Scale=120B2026.02 | 59.77 | 70.51 | 26.36 | 79.46 | 39.58 | 22.16 | 71.03 | 4.2 | 2.61 | |
| gemini-2.5-flashTraining Paradigm=Zero-Shot Prompting2026.02 | 58.9 | 70.1 | 22.97 | 81.64 | 35.85 | 19.66 | 75.18 | 3.31 | 1.85 | |
| Qwen3-4B-GRPOTraining Paradigm=On-Policy RL, Post-training Method=GRPO, Model Scale=4B2026.02 | 58.1 | 72.17 | 24.1 | 75.3 | 36.51 | 19.28 | 71.4 | 4.71 | 4.62 | |
| DeepSeek-R1-0528Training Paradigm=Zero-Shot Prompting2026.02 | 57.97 | 71.42 | 25.03 | 73.36 | 37.32 | 21.55 | 69.37 | 3.48 | 5.61 | |
| o4-miniTraining Paradigm=Zero-Shot Prompting2026.02 | 57.45 | 66.09 | 18.19 | 84.69 | 29.94 | 15.98 | 80.74 | 2.21 | 1.08 | |
| Qwen3-235B-A22BTraining Paradigm=Zero-Shot Prompting, Model Scale=235B2026.02 | 56.64 | 68.17 | 18.46 | 78.06 | 29.86 | 16.26 | 78.55 | 2.21 | 2.98 | |
| Qwen3-4B-DPOTraining Paradigm=Off-Policy Optimization, Post-training Method=DPO, Model Scale=4B2026.02 | 56.41 | 70.41 | 19.84 | 73.9 | 31.28 | 16.94 | 76.05 | 2.89 | 4.11 | |
| Qwen3-4B-SFTTraining Paradigm=Off-Policy Optimization, Post-training Method=SFT, Model Scale=4B2026.02 | 55.99 | 67.77 | 17.34 | 76.4 | 28.26 | 14.95 | 79.7 | 2.39 | 2.97 | |
| Qwen3-4B-ORPOTraining Paradigm=Off-Policy Optimization, Post-training Method=ORPO, Model Scale=4B2026.02 | 55.31 | 68.73 | 18.07 | 70.81 | 28.79 | 15.13 | 77.42 | 2.94 | 4.51 | |
| GPT-OSS-20BTraining Paradigm=Zero-Shot Prompting, Model Scale=20B2026.02 | 55.24 | 63.35 | 15.88 | 74.64 | 26.18 | 12.98 | 81.62 | 2.89 | 2.5 | |
| DeepSeek-R1-0528-Qwen3-8BTraining Paradigm=Zero-Shot Prompting, Model Scale=8B2026.02 | 53.97 | 66.55 | 13.35 | 71.18 | 22.48 | 11.78 | 82.82 | 1.57 | 3.83 | |
| Qwen3-4B-MARCOTraining Paradigm=On-Policy RL, Post-training Method=MARCO, Model Scale=4B2026.02 | 53.27 | 67.21 | 13.78 | 65.54 | 22.77 | 12.04 | 80.71 | 1.74 | 5.51 | |
| Qwen3-4BTraining Paradigm=Zero-Shot Prompting, Model Scale=4B2026.02 | 51.85 | 61.68 | 8.63 | 63.61 | 15.2 | 7.45 | 87.61 | 1.18 | 3.76 | |
| VulnLLM-R-7BTraining Paradigm=Zero-Shot Prompting, Model Scale=7B2026.02 | 50.76 | 61.42 | 6.83 | 56.22 | 12.18 | 5.08 | 89.61 | 1.75 | 3.57 | |
| Qwen3-4B-R2VULTraining Paradigm=Off-Policy Optimization, Post-training Method=R2VUL, Model Scale=4B2026.02 | 48.7 | 64.31 | 7.17 | 42.32 | 12.26 | 5.36 | 84.87 | 1.81 | 7.96 | |
| R2VUL-7BTraining Paradigm=Zero-Shot Prompting, Model Scale=7B2026.02 | 47.11 | 59.7 | 9.1 | 37.94 | 14.68 | 3.78 | 81.33 | 5.32 | 9.57 | |
| Qwen3-4B-ReVDTraining Paradigm=Off-Policy Optimization, Post-training Method=ReVD, Model Scale=4B2026.02 | 44.91 | 64.16 | 7.01 | 28.96 | 11.28 | 4.63 | 78.19 | 2.37 | 14.81 |