# Vulnerability Reasoning CWE-Guided Prompt (test)

Source: [Vulnerability Reasoning CWE-Guided Prompt (test)](<https://www.sota2.com/research/sota/vulnerability-reasoning-cwe-guided-prompt-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:31:15.371Z

Evaluation of model performance on vulnerability reasoning using the test split of the Vulnerability Reasoning CWE-Guided Prompt dataset.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Vulnerability Reasoning |
| Dataset (including any stated split) | Vulnerability Reasoning CWE-Guided Prompt (test) |
| Metric | Accuracy |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 36 of 36 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Accuracy](<https://www.sota2.com/research/sota/vulnerability-reasoning-cwe-guided-prompt-test.md?metric=Accuracy>): Higher is better; unit: PERCENT.

- [Precision](<https://www.sota2.com/research/sota/vulnerability-reasoning-cwe-guided-prompt-test.md?metric=Precision>): Higher is better; unit: PERCENT.

- [Recall](<https://www.sota2.com/research/sota/vulnerability-reasoning-cwe-guided-prompt-test.md?metric=Recall>): Higher is better; unit: PERCENT.

- [F1 Score](<https://www.sota2.com/research/sota/vulnerability-reasoning-cwe-guided-prompt-test.md?metric=F1%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Accuracy | Result date | Report paper |
| --- | --- | --- | --- |
| DAGVUL | 77.19 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 76.92 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 76.32 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| DAGVUL | 74.34 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| SFT | 73.68 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 72.63 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 71.23 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 71.05 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 71.05 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 71.03 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 70.54 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 70.18 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 70.18 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 69.91 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 69.33 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| SFT | 69.3 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 69.03 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 68.14 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 67.54 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 67.16 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| ORPO | 64.91 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 64.6 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| SFT | 64.04 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 61.61 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 61.06 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 60.18 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| ORPO | 59.65 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| SFT | 58.77 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 50 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 50 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| RAG | 50 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 49.56 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 49.12 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 49.12 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 48.04 | Not recorded | 698969f5c97a0b3e3b5dc46f |
| Baseline | 41.51 | Not recorded | 698969f5c97a0b3e3b5dc46f |

### DAGVUL

Accuracy: 77.19. Result ID: 6996dc845698665ec428f85f.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### RAG

Accuracy: 76.92. Result ID: 6996dc845698665ec428f872.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"Qwen3-30B-Reasoning","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### RAG

Accuracy: 76.32. Result ID: 6996dc845698665ec428f87a.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"Claude-Sonnet-4.5","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### DAGVUL

Accuracy: 74.34. Result ID: 6996dc845698665ec428f86b.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### SFT

Accuracy: 73.68. Result ID: 6996dc845698665ec428f85e.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b351e209131cea2ce7e.md>)

[Implementation](<https://github.com/hendrycks/apps>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 72.63. Result ID: 6996dc845698665ec428f871.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-30B-Reasoning","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 71.23. Result ID: 6996dc845698665ec428f873.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-30B-Reasoning","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 71.05. Result ID: 6996dc845698665ec428f860.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B-Reasoning","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 71.05. Result ID: 6996dc845698665ec428f861.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B-Reasoning","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 71.03. Result ID: 6996dc845698665ec428f870.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-30B-Reasoning","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### RAG

Accuracy: 70.54. Result ID: 6996dc845698665ec428f876.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"GPT-OSS-20B-High","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### RAG

Accuracy: 70.18. Result ID: 6996dc845698665ec428f862.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B-Reasoning","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 70.18. Result ID: 6996dc845698665ec428f878.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Claude-Sonnet-4.5","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 69.91. Result ID: 6996dc845698665ec428f858.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 69.33. Result ID: 6996dc845698665ec428f874.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"GPT-OSS-20B-High","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### SFT

Accuracy: 69.3. Result ID: 6996dc845698665ec428f86a.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b351e209131cea2ce7e.md>)

[Implementation](<https://github.com/hendrycks/apps>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 69.03. Result ID: 6996dc845698665ec428f879.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Claude-Sonnet-4.5","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 68.14. Result ID: 6996dc845698665ec428f863.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B-Reasoning","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 67.54. Result ID: 6996dc845698665ec428f87b.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Claude-Sonnet-4.5","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 67.16. Result ID: 6996dc845698665ec428f877.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"GPT-OSS-20B-High","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### ORPO

Accuracy: 64.91. Result ID: 6996dc845698665ec428f85c.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72851e209131cea56fe3.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### RAG

Accuracy: 64.6. Result ID: 6996dc845698665ec428f85a.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### SFT

Accuracy: 64.04. Result ID: 6996dc845698665ec428f85b.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b351e209131cea2ce7e.md>)

[Implementation](<https://github.com/hendrycks/apps>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 61.61. Result ID: 6996dc845698665ec428f875.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"GPT-OSS-20B-High","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 61.06. Result ID: 6996dc845698665ec428f859.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 60.18. Result ID: 6996dc845698665ec428f85d.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Qwen3-8B","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### ORPO

Accuracy: 59.65. Result ID: 6996dc845698665ec428f868.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72851e209131cea56fe3.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### SFT

Accuracy: 58.77. Result ID: 6996dc845698665ec428f867.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b351e209131cea2ce7e.md>)

[Implementation](<https://github.com/hendrycks/apps>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 50. Result ID: 6996dc845698665ec428f864.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### RAG

Accuracy: 50. Result ID: 6996dc845698665ec428f866.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### RAG

Accuracy: 50. Result ID: 6996dc845698665ec428f86e.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B-Reasoning","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 49.56. Result ID: 6996dc845698665ec428f86d.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B-Reasoning","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 49.12. Result ID: 6996dc845698665ec428f865.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"CoT"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 49.12. Result ID: 6996dc845698665ec428f86c.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B-Reasoning","Inference Strategy":"Simple prompting"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 48.04. Result ID: 6996dc845698665ec428f86f.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B-Reasoning","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

### Baseline

Accuracy: 41.51. Result ID: 6996dc845698665ec428f869.

[Reporting paper](<https://www.sota2.com/research/paper/698969f5c97a0b3e3b5dc46f.md>)

Recorded evaluation configuration: {"Base Model":"Llama3.1-Nemo-8B","Inference Strategy":"DAG"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.