# PolicyBank: Evolving Policy Understanding for LLM Agents

Source: [PolicyBank: Evolving Policy Understanding for LLM Agents](<https://www.sota2.com/research/paper/69e5b30a7d0172140ec7dfa4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-21T05:00:04.784Z

Authors: Jihye Choi, Jinsung Yoon, Long T. Le, Somesh Jha, Tomas Pfister

Published: 2026-04-16T20:29:30.000Z

## Abstract

LLM agents operating under organizational policies must comply with authorization constraints typically specified in natural language. In practice, such specifications inevitably contain ambiguities and logical or semantic gaps that cause the agent's behavior to systematically diverge from the true requirements. We ask: by letting an agent evolve its policy understanding through interaction and corrective feedback from pre-deployment testing, can it autonomously refine its interpretation to close specification gaps? We propose PolicyBank, a memory mechanism that maintains structured, tool-level policy insights and iteratively refines them -- unlike existing memory mechanisms that treat the policy as immutable ground truth, reinforcing "compliant but wrong" behaviors. We also contribute a systematic testbed by extending a popular tool-calling benchmark with controlled policy gaps that isolate alignment failures from execution failures. While existing memory mechanisms achieve near-zero success on policy-gap scenarios, PolicyBank closes up to 82% of the gap toward a human oracle.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Sister Tasks on extended tau-Bench Airline domain](<https://www.sota2.com/research/sota/sister-tasks-on-extended-tau-bench-airline-domain.md>): Sister Tasks; extended tau-Bench Airline domain; Pass@1 Rate: 74.

- [Sister Tasks on extended tau-Bench Retail domain](<https://www.sota2.com/research/sota/sister-tasks-on-extended-tau-bench-retail-domain.md>): Sister Tasks; extended tau-Bench Retail domain; Pass@1 Rate: 83.

- [Original Tasks on tau-Bench Airline domain extended](<https://www.sota2.com/research/sota/original-tasks-on-tau-bench-airline-domain-extended.md>): Original Tasks; tau-Bench Airline domain extended; Pass Rate @ Threshold 1: 70.

- [Original Tasks on extended tau-Bench Retail domain](<https://www.sota2.com/research/sota/original-tasks-on-extended-tau-bench-retail-domain.md>): Original Tasks; extended tau-Bench Retail domain; Pass@1: 89.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.15505>)

[PDF](<https://arxiv.org/pdf/2604.15505.pdf>)