Vulnerability Detection on CASTLE 250-sample benchmark 1.0 benchmark leaderboard · SOTA2 Research

Vulnerability Detection on CASTLE 250-sample benchmark 1.0

1,068CASTLE ScoreVulnScout-C

Evaluation Results

Method	Links
VulnScout-C2026.03		1,068	136	77	16	30
GPT-o3 Mini2026.03		977	126	60	73	36
GPT-o12026.03		962	128	56	90	35
DeepSeek R12026.03		956	148	41	166	17
GPT-4o2026.03		954	136	45	116	43
GPT-4o Mini2026.03		761	134	27	263	43
QWEN 2.5CI (32B)2026.03		708	114	31	224	49
ESBMC2026.03		661	53	91	32	97
CodeQL2026.03		634	45	79	49	112
GCC Fanalyzer2026.03		559	41	76	93	109
Snyk2026.03		552	26	82	42	124
Semgrep Code2026.03		541	36	73	70	120
CBMC2026.03		536	18	100	0	132
Falcon 3 (7B)2026.03		521	30	76	76	124
SonarQube2026.03		511	43	68	135	107
Aikido2026.03		481	14	83	40	136
Jit2026.03		478	21	78	68	134
Mistral Ins. (7B)2026.03		446	63	23	215	91
Gemma 2 (9B)2026.03		436	63	42	258	95
Coverity2026.03		425	31	86	62	119
LLAMA 3.1 (8B)2026.03		417	83	22	337	80
Cppcheck2026.03		406	19	100	9	131
Clang Analyzer2026.03		381	13	99	2	137
GitLab SAST2026.03		374	36	67	240	120
VulBERTa2026.03		-12	103	37	130	12
GraphCodeBERT2026.03		-116	146	106	189	3
CodeBERT2026.03		-166	145	217	241	0
Splint2026.03		-598	23	36	1,027	127
CodeThreat2026.03		-692	24	2	1,101	126