# Safeguarding LLM Agents from Misalignment through Provenance Analysis Source: [Safeguarding LLM Agents from Misalignment through Provenance Analysis]() Collection index: [llms.txt]() Data updated: 2026-08-12T05:00:21.403Z Authors: Yining She, Yiliang Liang, Eunsuk Kang Published: 2026-05-01T03:16:35.000Z ## Abstract As LLM agents gain increasing access to powerful tools, ensuring that their actions align with the user's intent becomes critical. When an agent's proposed action deviates from that intent---a phenomenon called misalignment---it may cause harm that is difficult to undo. Existing runtime guardrails rely on an LLM-as-a-judge paradigm that lacks a systematic framework for reasoning about alignment, often producing inconsistent or difficult-to-audit judgments. Motivated by provenance analysis, we propose a conceptual framework that formalizes misalignment detection as determining whether a proposed tool call is supported by traceable evidence in the agent's context. Based on this framework, we build ProvenanceGuard, a multi-stage pipeline that analyzes the agent's action for three types of misalignment before its execution and only allows aligned actions. We evaluated ProvenanceGuard on AgentSafetyBench and WorkBench, across 11 backbone LLMs. Compared to the LLM-as-a-judge baseline, ProvenanceGuard reduces error rate on misaligned traces from 44.3% to 2.1% on Agent-SafetyBench and from 32.4% to 18.7% on WorkBench, while reducing interventions on task-successful traces from 31.2% to 13.0% and introducing no statistically significant increase in unnecessary interventions on aligned traces. These results demonstrate that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents from misalignment. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Misalignment Detection on Workbench n=62 (general)](): Misalignment Detection; Workbench n=62 (general); Error Rate (ER): 0.3. - [Misalignment Detection on Agent-SafetyBench underspecified queries n=148](): Misalignment Detection; Agent-SafetyBench underspecified queries n=148; Error Rate (ER): 1.1. - [Misalignment Detection on Agent-SafetyBench n=35 (aligned traces)](): Misalignment Detection; Agent-SafetyBench n=35 (aligned traces); Avg. IR: 18.9. - [Misalignment Detection on WorkBench (n=326)](): Misalignment Detection; WorkBench (n=326); Average IR: 18.1. ## Original sources and implementation [Original paper]() [PDF]()