Vulnerability Reasoning CWE-Guided Prompt (test)
77.19AccuracyDAGVUL
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DAGVULBase Model=Qwen3-8B, Inference Strategy=DAG2026.02 | 77.19 | 81.63 | 70.18 | 75.47 | |
| RAGBase Model=Qwen3-30B-Reasoning, Inference Strategy=CoT2026.02 | 76.92 | 87.18 | 64.15 | 73.91 | |
| RAGBase Model=Claude-Sonnet-4.5, Inference Strategy=CoT2026.02 | 76.32 | 76.79 | 75.44 | 76.11 | |
| DAGVULBase Model=Llama3.1-Nemo-8B, Inference Strategy=DAG2026.02 | 74.34 | 78 | 68.42 | 72.9 | |
| SFTBase Model=Qwen3-8B, Inference Strategy=DAG2026.02 | 73.68 | 81.4 | 61.4 | 70 | |
| BaselineBase Model=Qwen3-30B-Reasoning, Inference Strategy=CoT2026.02 | 72.63 | 84.38 | 56.25 | 67.5 | |
| BaselineBase Model=Qwen3-30B-Reasoning, Inference Strategy=DAG2026.02 | 71.23 | 76.92 | 57.14 | 65.57 | |
| BaselineBase Model=Qwen3-8B-Reasoning, Inference Strategy=Simple prompting2026.02 | 71.05 | 76.09 | 61.4 | 67.96 | |
| BaselineBase Model=Qwen3-8B-Reasoning, Inference Strategy=CoT2026.02 | 71.05 | 74 | 64.91 | 69.16 | |
| BaselineBase Model=Qwen3-30B-Reasoning, Inference Strategy=Simple prompting2026.02 | 71.03 | 78.05 | 59.26 | 67.37 | |
| RAGBase Model=GPT-OSS-20B-High, Inference Strategy=CoT2026.02 | 70.54 | 73.47 | 64.29 | 68.57 | |
| RAGBase Model=Qwen3-8B-Reasoning, Inference Strategy=CoT2026.02 | 70.18 | 71.7 | 66.67 | 69.09 | |
| BaselineBase Model=Claude-Sonnet-4.5, Inference Strategy=Simple prompting2026.02 | 70.18 | 74.47 | 61.4 | 67.31 | |
| BaselineBase Model=Qwen3-8B, Inference Strategy=Simple prompting2026.02 | 69.91 | 92.31 | 42.86 | 58.54 | |
| BaselineBase Model=GPT-OSS-20B-High, Inference Strategy=Simple prompting2026.02 | 69.33 | 66.67 | 51.61 | 58.18 | |
| SFTBase Model=Llama3.1-Nemo-8B, Inference Strategy=DAG2026.02 | 69.3 | 70.37 | 66.67 | 68.47 | |
| BaselineBase Model=Claude-Sonnet-4.5, Inference Strategy=CoT2026.02 | 69.03 | 66.15 | 76.79 | 71.07 | |
| BaselineBase Model=Qwen3-8B-Reasoning, Inference Strategy=DAG2026.02 | 68.14 | 67.24 | 69.64 | 68.42 | |
| BaselineBase Model=Claude-Sonnet-4.5, Inference Strategy=DAG2026.02 | 67.54 | 66.67 | 70.18 | 68.38 | |
| BaselineBase Model=GPT-OSS-20B-High, Inference Strategy=DAG2026.02 | 67.16 | 71.43 | 58.82 | 64.52 | |
| ORPOBase Model=Qwen3-8B, Inference Strategy=CoT2026.02 | 64.91 | 61.97 | 77.19 | 68.75 | |
| RAGBase Model=Qwen3-8B, Inference Strategy=CoT2026.02 | 64.6 | 62.32 | 75.44 | 68.25 | |
| SFTBase Model=Qwen3-8B, Inference Strategy=CoT2026.02 | 64.04 | 61.11 | 77.19 | 68.22 | |
| BaselineBase Model=GPT-OSS-20B-High, Inference Strategy=CoT2026.02 | 61.61 | 63.27 | 55.36 | 59.05 | |
| BaselineBase Model=Qwen3-8B, Inference Strategy=CoT2026.02 | 61.06 | 59.09 | 69.64 | 63.93 | |
| BaselineBase Model=Qwen3-8B, Inference Strategy=DAG2026.02 | 60.18 | 57.32 | 82.46 | 67.63 | |
| ORPOBase Model=Llama3.1-Nemo-8B, Inference Strategy=CoT2026.02 | 59.65 | 57.33 | 75.44 | 65.15 | |
| SFTBase Model=Llama3.1-Nemo-8B, Inference Strategy=CoT2026.02 | 58.77 | 56.58 | 75.44 | 64.66 | |
| BaselineBase Model=Llama3.1-Nemo-8B, Inference Strategy=Simple prompting2026.02 | 50 | 50 | 57.89 | 53.66 | |
| RAGBase Model=Llama3.1-Nemo-8B, Inference Strategy=CoT2026.02 | 50 | 50 | 57.89 | 53.66 | |
| RAGBase Model=Llama3.1-Nemo-8B-Reasoning, Inference Strategy=CoT2026.02 | 50 | 50 | 59.65 | 54.4 | |
| BaselineBase Model=Llama3.1-Nemo-8B-Reasoning, Inference Strategy=CoT2026.02 | 49.56 | 49.21 | 55.36 | 52.1 | |
| BaselineBase Model=Llama3.1-Nemo-8B, Inference Strategy=CoT2026.02 | 49.12 | 49.21 | 54.39 | 51.67 | |
| BaselineBase Model=Llama3.1-Nemo-8B-Reasoning, Inference Strategy=Simple prompting2026.02 | 49.12 | 49.25 | 57.89 | 53.23 | |
| BaselineBase Model=Llama3.1-Nemo-8B-Reasoning, Inference Strategy=DAG2026.02 | 48.04 | 48.53 | 64.71 | 55.46 | |
| BaselineBase Model=Llama3.1-Nemo-8B, Inference Strategy=DAG2026.02 | 41.51 | 44 | 62.26 | 51.56 |