Vulnerability Detection on PrimeVul (test)
68.74F1 ScoreReVD
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ReVDBackbone=StarCoder2-7B2025.06 | 68.74 | — | — | 55.63 | 11.26 | |
| CPRVULBackbone=Qwen2.5-32B2026.02 | 67.93 | 67.62 | 68.23 | 67.78 | — | |
| ReVDBackbone=Qwen2.5-Coder-7B-Instruct2025.06 | 63.83 | — | — | 58.05 | 16.09 | |
| GPTLens2026.02 | 57.63 | 51.44 | 65.52 | 51.84 | — | |
| VulTrial2026.02 | 56.18 | 54.95 | 57.47 | 55.17 | — | |
| Llama3.1-70B-InstructType=COT2025.06 | 55.9 | — | — | 49.77 | -0.46 | |
| ReVDBackbone=Llama-3.1-8B-Instruct2025.06 | 54.91 | — | — | 56.21 | 12.41 | |
| GPT-4Type=COT2025.06 | 43.4 | — | — | 51.72 | 3.44 | |
| LLMxCPG2026.02 | 41.87 | 50.51 | 35.75 | 50.36 | — | |
| Llama-3.1-8B-InstructType=COT2025.06 | 40.58 | — | — | 48.16 | -3.68 | |
| Qwen2.5-Coder-7B-InstructType=COT2025.06 | 29.86 | — | — | 49.77 | -0.46 | |
| Ding et al.'s CoT2026.02 | 28.14 | 53.55 | 19.08 | 51.26 | — | |
| Qwen2.5-32b-Coder-InstructType=COT2025.06 | 26.64 | — | — | 50 | 0 | |
| ExtensionTarget DLVD Model=Linevul, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 23.22 | 18.51 | 31.15 | — | — | |
| NoAugTarget DLVD Model=Linevul, Augmentation Samples=02024.08 | 23.06 | 18.31 | 31.15 | — | — | |
| VGXTarget DLVD Model=Linevul, Augmentation Samples=5K2024.08 | 23.06 | 19.77 | 27.69 | — | — | |
| VulGenTarget DLVD Model=Linevul, Augmentation Samples=5K2024.08 | 22.99 | 19.39 | 28.23 | — | — | |
| ROSTarget DLVD Model=Linevul, Augmentation Samples=5K2024.08 | 22.89 | 18.48 | 30.05 | — | — | |
| InjectionTarget DLVD Model=Linevul, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 22.42 | 17.29 | 31.88 | — | — | |
| MutationTarget DLVD Model=Linevul, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 19.85 | 15.71 | 26.96 | — | — | |
| LineVulType=SFT2025.06 | 17.7 | — | — | 49.77 | -0.46 | |
| UniXCoderType=SFT2025.06 | 16.48 | — | — | 49.89 | -0.23 | |
| CodeBERTType=SFT2025.06 | 16.47 | — | — | 51.03 | 2.07 | |
| ExtensionTarget DLVD Model=Reveal, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 14.11 | 8.51 | 41.22 | — | — | |
| InjectionTarget DLVD Model=Reveal, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 13.51 | 7.81 | 49.76 | — | — | |
| ExtensionTarget DLVD Model=Devign, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 13.33 | 7.61 | 53.41 | — | — | |
| InjectionTarget DLVD Model=Devign, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 13.31 | 7.57 | 55.37 | — | — | |
| MutationTarget DLVD Model=Reveal, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 13.25 | 7.81 | 43.66 | — | — | |
| ROSTarget DLVD Model=Reveal, Augmentation Samples=5K2024.08 | 13.07 | 7.58 | 43.66 | — | — | |
| ROSTarget DLVD Model=Devign, Augmentation Samples=5K2024.08 | 13.04 | 7.45 | 52.44 | — | — | |
| MutationTarget DLVD Model=Devign, Augmentation Samples=5K, LLM used for augmentation=GPT-4o-mini2024.08 | 12.63 | 7.17 | 52.93 | — | — | |
| NoAugTarget DLVD Model=Reveal, Augmentation Samples=02024.08 | 12.62 | 7.35 | 44.63 | — | — | |
| VGXTarget DLVD Model=Devign, Augmentation Samples=5K2024.08 | 10.15 | 5.61 | 53.66 | — | — | |
| VGXTarget DLVD Model=Reveal, Augmentation Samples=5K2024.08 | 9.82 | 5.44 | 50.73 | — | — | |
| VulGenTarget DLVD Model=Reveal, Augmentation Samples=5K2024.08 | 9.72 | 5.29 | 59.27 | — | — | |
| VulGenTarget DLVD Model=Devign, Augmentation Samples=5K2024.08 | 9.11 | 4.94 | 59.76 | — | — | |
| NoAugTarget DLVD Model=Devign, Augmentation Samples=02024.08 | 8.55 | 25.93 | 5.12 | — | — | |
| StarCoder2-7BType=COT2025.06 | 0.91 | — | — | 50.11 | 0.23 |