# Vulnerability Detection on CASTLE 250-sample benchmark 1.0 Source: [Vulnerability Detection on CASTLE 250-sample benchmark 1.0]() Collection index: [llms.txt]() Data updated: 2026-03-31T12:02:16.774Z Evaluation of vulnerability detection performance using the CASTLE 250-sample benchmark version 1.0, measured by CASTLE Score, True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Vulnerability Detection | | Dataset (including any stated split) | CASTLE 250-sample benchmark 1.0 | | Metric | CASTLE Score | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 29 of 29 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [CASTLE Score](): Higher is better; unit: SCALAR. - [True Positives (TP)](): Higher is better; unit: SCALAR. - [True Negatives (TN)](): Higher is better; unit: SCALAR. - [False Positives (FP)](): Lower is better; unit: SCALAR. - [False Negatives (FN)](): Lower is better; unit: SCALAR. ## Recorded results | Method | CASTLE Score | Result date | Report paper | | --- | --- | --- | --- | | VulnScout-C | 1068 | Not recorded | 69cb55760636e5bf4afca347 | | GPT-o3 Mini | 977 | Not recorded | 69cb55760636e5bf4afca347 | | GPT-o1 | 962 | Not recorded | 69cb55760636e5bf4afca347 | | DeepSeek R1 | 956 | Not recorded | 69cb55760636e5bf4afca347 | | GPT-4o | 954 | Not recorded | 69cb55760636e5bf4afca347 | | GPT-4o Mini | 761 | Not recorded | 69cb55760636e5bf4afca347 | | QWEN 2.5CI (32B) | 708 | Not recorded | 69cb55760636e5bf4afca347 | | ESBMC | 661 | Not recorded | 69cb55760636e5bf4afca347 | | CodeQL | 634 | Not recorded | 69cb55760636e5bf4afca347 | | GCC Fanalyzer | 559 | Not recorded | 69cb55760636e5bf4afca347 | | Snyk | 552 | Not recorded | 69cb55760636e5bf4afca347 | | Semgrep Code | 541 | Not recorded | 69cb55760636e5bf4afca347 | | CBMC | 536 | Not recorded | 69cb55760636e5bf4afca347 | | Falcon 3 (7B) | 521 | Not recorded | 69cb55760636e5bf4afca347 | | SonarQube | 511 | Not recorded | 69cb55760636e5bf4afca347 | | Aikido | 481 | Not recorded | 69cb55760636e5bf4afca347 | | Jit | 478 | Not recorded | 69cb55760636e5bf4afca347 | | Mistral Ins. (7B) | 446 | Not recorded | 69cb55760636e5bf4afca347 | | Gemma 2 (9B) | 436 | Not recorded | 69cb55760636e5bf4afca347 | | Coverity | 425 | Not recorded | 69cb55760636e5bf4afca347 | | LLAMA 3.1 (8B) | 417 | Not recorded | 69cb55760636e5bf4afca347 | | Cppcheck | 406 | Not recorded | 69cb55760636e5bf4afca347 | | Clang Analyzer | 381 | Not recorded | 69cb55760636e5bf4afca347 | | GitLab SAST | 374 | Not recorded | 69cb55760636e5bf4afca347 | | VulBERTa | -12 | Not recorded | 69cb55760636e5bf4afca347 | | GraphCodeBERT | -116 | Not recorded | 69cb55760636e5bf4afca347 | | CodeBERT | -166 | Not recorded | 69cb55760636e5bf4afca347 | | Splint | -598 | Not recorded | 69cb55760636e5bf4afca347 | | CodeThreat | -692 | Not recorded | 69cb55760636e5bf4afca347 | ### VulnScout-C CASTLE Score: 1068. Result ID: 69cbb7c747f34ce924b52d56. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Sparse MoE Encoder Architecture","Context Window":"Extended"}. This record does not assert independent reproduction. ### GPT-o3 Mini CASTLE Score: 977. Result ID: 69cbb7c747f34ce924b52d55. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### GPT-o1 CASTLE Score: 962. Result ID: 69cbb7c747f34ce924b52d54. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### DeepSeek R1 CASTLE Score: 956. Result ID: 69cbb7c747f34ce924b52d53. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### GPT-4o CASTLE Score: 954. Result ID: 69cbb7c747f34ce924b52d52. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### GPT-4o Mini CASTLE Score: 761. Result ID: 69cbb7c747f34ce924b52d50. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### QWEN 2.5CI (32B) CASTLE Score: 708. Result ID: 69cbb7c747f34ce924b52d51. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### ESBMC CASTLE Score: 661. Result ID: 69cbb7c747f34ce924b52d4b. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Formal Verification / Static Analysis Tool","Version":"7.8.1"}. This record does not assert independent reproduction. ### CodeQL CASTLE Score: 634. Result ID: 69cbb7c747f34ce924b52d4a. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"2.20.1"}. This record does not assert independent reproduction. ### GCC Fanalyzer CASTLE Score: 559. Result ID: 69cbb7c747f34ce924b52d49. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"13.3.0"}. This record does not assert independent reproduction. ### Snyk CASTLE Score: 552. Result ID: 69cbb7c747f34ce924b52d48. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"1.1295.4"}. This record does not assert independent reproduction. ### Semgrep Code CASTLE Score: 541. Result ID: 69cbb7c747f34ce924b52d47. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"1.110.0"}. This record does not assert independent reproduction. ### CBMC CASTLE Score: 536. Result ID: 69cbb7c747f34ce924b52d46. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Formal Verification / Static Analysis Tool","Version":"5.95.1"}. This record does not assert independent reproduction. ### Falcon 3 (7B) CASTLE Score: 521. Result ID: 69cbb7c747f34ce924b52d4f. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### SonarQube CASTLE Score: 511. Result ID: 69cbb7c747f34ce924b52d45. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"25.3.0"}. This record does not assert independent reproduction. ### Aikido CASTLE Score: 481. Result ID: 69cbb7c747f34ce924b52d43. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool"}. This record does not assert independent reproduction. ### Jit CASTLE Score: 478. Result ID: 69cbb7c747f34ce924b52d44. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool"}. This record does not assert independent reproduction. ### Mistral Ins. (7B) CASTLE Score: 446. Result ID: 69cbb7c747f34ce924b52d4e. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### Gemma 2 (9B) CASTLE Score: 436. Result ID: 69cbb7c747f34ce924b52d4d. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### Coverity CASTLE Score: 425. Result ID: 69cbb7c747f34ce924b52d42. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"2024.6.1"}. This record does not assert independent reproduction. ### LLAMA 3.1 (8B) CASTLE Score: 417. Result ID: 69cbb7c747f34ce924b52d4c. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Large Language Model"}. This record does not assert independent reproduction. ### Cppcheck CASTLE Score: 406. Result ID: 69cbb7c747f34ce924b52d41. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"2.13.0"}. This record does not assert independent reproduction. ### Clang Analyzer CASTLE Score: 381. Result ID: 69cbb7c747f34ce924b52d3f. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"18.1.3"}. This record does not assert independent reproduction. ### GitLab SAST CASTLE Score: 374. Result ID: 69cbb7c747f34ce924b52d40. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"15.2.1"}. This record does not assert independent reproduction. ### VulBERTa CASTLE Score: -12. Result ID: 69cbb7c747f34ce924b52d3c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Fine-tuned Encoder Baseline","Fine-tuned":"true"}. This record does not assert independent reproduction. ### GraphCodeBERT CASTLE Score: -116. Result ID: 69cbb7c747f34ce924b52d3b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Fine-tuned Encoder Baseline","Fine-tuned":"true"}. This record does not assert independent reproduction. ### CodeBERT CASTLE Score: -166. Result ID: 69cbb7c747f34ce924b52d3a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Fine-tuned Encoder Baseline","Fine-tuned":"true"}. This record does not assert independent reproduction. ### Splint CASTLE Score: -598. Result ID: 69cbb7c747f34ce924b52d3e. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool","Version":"3.1.2"}. This record does not assert independent reproduction. ### CodeThreat CASTLE Score: -692. Result ID: 69cbb7c747f34ce924b52d3d. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Static Analysis Tool"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.