Vulnerability Detection on PrimeVul Paired (test)
275Pair-Correct CountPhoenix
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| PhoenixModel=14B+9B, Category=Ours (Zero-Shot, Open-Source)2026.04 | 275 | 82.5 | 79.2 | 86 | — | — | — | — | — | — | |
| VulTriageBackbone=GPT-4o2026.05 | 147 | 69.07 | 60 | 81.38 | 207 | 52 | 29 | 288 | 54.25 | 63.56 | |
| VulTrial+FTModel=GPT-4o+FT, Category=Multi-Agent2026.04 | 96 | — | — | — | — | — | — | — | — | — | |
| RASM-VulModel=DeepSeek-V3, Category=Retrieval-Augmented2026.04 | 93 | 66.8 | 57.2 | 80.2 | — | — | — | — | — | — | |
| Phoenix (Blind)Model=Qwen3.5-9B, Category=Ours (Zero-Shot, Open-Source)2026.04 | 85 | 51 | 59.9 | 44.4 | — | — | — | — | — | — | |
| VulTrialModel=GPT-4o x4, Category=Multi-Agent2026.04 | 81 | 56.3 | 53 | 60 | — | — | — | — | — | — | |
| VulTrialBackbone=GPT-4o2026.05 | 81 | 56.28 | 53.17 | 59.77 | 179 | 125 | 50 | 354 | 52.64 | 53.56 | |
| DeepSeek-V3Model=671B MoE, Category=Zero-Shot LLM (single-agent)2026.04 | 67 | 46.9 | 50.5 | 43.8 | — | — | — | — | — | — | |
| Qwen2.5-72BModel=72B, Category=Zero-Shot LLM (single-agent)2026.04 | 49 | 56.9 | 52.8 | 61.8 | — | — | — | — | — | — | |
| GPTLensModel=GPT-4o, Category=Zero-Shot LLM (single-agent)2026.04 | 44 | — | 51 | 65 | — | — | — | — | — | — | |
| GPTLensBackbone=GPT-4o2026.05 | 44 | 57.63 | 51.44 | 65.52 | 241 | 122 | 28 | 391 | 61.84 | 51.84 | |
| CoTModel=GPT-4o, Category=Zero-Shot LLM (single-agent)2026.04 | 40 | — | 53 | 19 | — | — | — | — | — | — | |
| Ding et al.’s CoTBackbone=GPT-4o, Prompting Strategy=best-performing prompting strategy reported in PrimeVul2026.05 | 40 | 28.14 | 53.55 | 19.08 | 43 | 323 | 29 | 395 | 16.55 | 51.26 | |
| ConCollBackbone=GPT-4o2026.05 | 22 | 62.87 | 50.78 | 82.53 | 337 | 65 | 11 | 413 | 80 | 51.26 | |
| LineVulCheckpoint=Released checkpoint from original study2026.05 | 7 | 30.82 | 51.06 | 22.07 | 89 | 336 | 3 | 428 | 21.15 | 50.46 | |
| UniXCoderTraining Set=Paired PrimeVul train/val subset2026.05 | 6 | 66.09 | 50.36 | 96.09 | 412 | 17 | 0 | 429 | 94.71 | 50.69 | |
| CodeBERTTraining Set=Full PrimeVul train/val set2026.05 | 5 | 16.12 | 48.84 | 9.66 | 37 | 386 | 7 | 430 | 10.11 | 49.77 | |
| UniXCoderTraining Set=Full PrimeVul train/val set2026.05 | 4 | 12.17 | 51.72 | 6.9 | 26 | 403 | 2 | 431 | 6.44 | 50.23 | |
| CodeBERTTraining Set=Paired PrimeVul train/val subset2026.05 | 0 | 66.67 | 50 | 100 | 435 | 0 | 0 | 435 | 100 | 50 | |
| CodeT5Training Set=Full PrimeVul train/val set2026.05 | 0 | 20.58 | 47.9 | 13.1 | 57 | 373 | 5 | 435 | 14.25 | 49.43 | |
| CodeT5Training Set=Paired PrimeVul train/val subset2026.05 | 0 | 66.62 | 50 | 99.77 | 434 | 1 | 0 | 435 | 99.77 | 50 | |
| LineVulTraining Set=Paired PrimeVul train/val subset2026.05 | 0 | 66.62 | 50 | 99.77 | 434 | 1 | 0 | 435 | 99.77 | 50 |