Vulnerability Detection on CASTLE 250-sample benchmark 1.0
1,068CASTLE ScoreVulnScout-C
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| VulnScout-CModel Category=Sparse MoE Encoder Architecture, Context Window=Extended2026.03 | 1,068 | 136 | 77 | 16 | 30 | |
| GPT-o3 MiniModel Category=Large Language Model2026.03 | 977 | 126 | 60 | 73 | 36 | |
| GPT-o1Model Category=Large Language Model2026.03 | 962 | 128 | 56 | 90 | 35 | |
| DeepSeek R1Model Category=Large Language Model2026.03 | 956 | 148 | 41 | 166 | 17 | |
| GPT-4oModel Category=Large Language Model2026.03 | 954 | 136 | 45 | 116 | 43 | |
| GPT-4o MiniModel Category=Large Language Model2026.03 | 761 | 134 | 27 | 263 | 43 | |
| QWEN 2.5CI (32B)Model Category=Large Language Model2026.03 | 708 | 114 | 31 | 224 | 49 | |
| ESBMCModel Category=Formal Verification / Static Analysis Tool, Version=7.8.12026.03 | 661 | 53 | 91 | 32 | 97 | |
| CodeQLModel Category=Static Analysis Tool, Version=2.20.12026.03 | 634 | 45 | 79 | 49 | 112 | |
| GCC FanalyzerModel Category=Static Analysis Tool, Version=13.3.02026.03 | 559 | 41 | 76 | 93 | 109 | |
| SnykModel Category=Static Analysis Tool, Version=1.1295.42026.03 | 552 | 26 | 82 | 42 | 124 | |
| Semgrep CodeModel Category=Static Analysis Tool, Version=1.110.02026.03 | 541 | 36 | 73 | 70 | 120 | |
| CBMCModel Category=Formal Verification / Static Analysis Tool, Version=5.95.12026.03 | 536 | 18 | 100 | 0 | 132 | |
| Falcon 3 (7B)Model Category=Large Language Model2026.03 | 521 | 30 | 76 | 76 | 124 | |
| SonarQubeModel Category=Static Analysis Tool, Version=25.3.02026.03 | 511 | 43 | 68 | 135 | 107 | |
| AikidoModel Category=Static Analysis Tool2026.03 | 481 | 14 | 83 | 40 | 136 | |
| JitModel Category=Static Analysis Tool2026.03 | 478 | 21 | 78 | 68 | 134 | |
| Mistral Ins. (7B)Model Category=Large Language Model2026.03 | 446 | 63 | 23 | 215 | 91 | |
| Gemma 2 (9B)Model Category=Large Language Model2026.03 | 436 | 63 | 42 | 258 | 95 | |
| CoverityModel Category=Static Analysis Tool, Version=2024.6.12026.03 | 425 | 31 | 86 | 62 | 119 | |
| LLAMA 3.1 (8B)Model Category=Large Language Model2026.03 | 417 | 83 | 22 | 337 | 80 | |
| CppcheckModel Category=Static Analysis Tool, Version=2.13.02026.03 | 406 | 19 | 100 | 9 | 131 | |
| Clang AnalyzerModel Category=Static Analysis Tool, Version=18.1.32026.03 | 381 | 13 | 99 | 2 | 137 | |
| GitLab SASTModel Category=Static Analysis Tool, Version=15.2.12026.03 | 374 | 36 | 67 | 240 | 120 | |
| VulBERTaModel Category=Fine-tuned Encoder Baseline, Fine-tuned=true2026.03 | -12 | 103 | 37 | 130 | 12 | |
| GraphCodeBERTModel Category=Fine-tuned Encoder Baseline, Fine-tuned=true2026.03 | -116 | 146 | 106 | 189 | 3 | |
| CodeBERTModel Category=Fine-tuned Encoder Baseline, Fine-tuned=true2026.03 | -166 | 145 | 217 | 241 | 0 | |
| SplintModel Category=Static Analysis Tool, Version=3.1.22026.03 | -598 | 23 | 36 | 1,027 | 127 | |
| CodeThreatModel Category=Static Analysis Tool2026.03 | -692 | 24 | 2 | 1,101 | 126 |