# Evaluating and Enhancing the Vulnerability Reasoning Capabilities of Large Language Models Source: [Evaluating and Enhancing the Vulnerability Reasoning Capabilities of Large Language Models]() Collection index: [llms.txt]() Data updated: 2026-02-10T05:00:04.897Z Authors: Li Lu, Yanjie Zhao, Hongzhou Rao, Kechi Zhang, Haoyu Wang Published: 2026-02-06T13:19:45.000Z ## Abstract Large Language Models (LLMs) have demonstrated remarkable proficiency in vulnerability detection. However, a critical reliability gap persists: models frequently yield correct detection verdicts based on hallucinated logic or superficial patterns that deviate from the actual root cause. This misalignment remains largely obscured because contemporary benchmarks predominantly prioritize coarse-grained classification metrics, lacking the granular ground truth required to evaluate the underlying reasoning process. To bridge this gap, we first construct a benchmark consisting of two datasets: (1) real-world vulnerabilities with expert-curated causal reasoning as ground truth, and (2) semantically equivalent code perturbations for assessing reasoning robustness. Our large-scale empirical study reveals that even state-of-the-art models struggle to maintain logical consistency during semantic code comprehension, exhibiting 12 systematic failure patterns. Addressing these limitations, we propose DAGVul, a novel framework that models vulnerability reasoning as a Directed Acyclic Graph (DAG) generation task. Unlike linear chain-of-thought (CoT), our approach explicitly maps causal dependencies to enforce structural consistency. By further introducing Reinforcement Learning with Verifiable Rewards (RLVR), we align model reasoning trace with program-intrinsic logic. Experimental results demonstrate that our framework improves the reasoning F1-score by an average of 18.9% over all the baselines. Remarkably, our 8B-parameter implementation not only outperforms existing models of comparable scale but also surpasses specialized large-scale reasoning models, including Qwen3-30B-Reasoning and GPT-OSS-20B-High. It is even competitive with state-of-the-art models like Claude-Sonnet-4.5 (75.47% vs. 76.11%), establishing new efficiency in vulnerability reasoning across model scales. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Vulnerability Reasoning on Vulnerability Reasoning Basic Prompt (test)](): Vulnerability Reasoning; Vulnerability Reasoning Basic Prompt (test); RAcc: 76.11. - [Vulnerability Reasoning CWE-Guided Prompt (test)](): Vulnerability Reasoning; Vulnerability Reasoning CWE-Guided Prompt (test); Accuracy: 77.19. ## Original sources and implementation [Original paper]() [PDF]()