AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
About
Large Language Models (LLMs) are increasingly used as autonomous agents in complex, long-horizon applications, where effective memory is critical for sustained performance. Yet existing memory benchmarks are largely dialogue-centric, while real agent memory consists of continuous agent-environment interaction trajectories composed of states, actions, observations, and tool outputs. To address this gap, we introduce **AMA-Bench** (**A**gent **M**emory with **A**ny length), a benchmark for evaluating long-horizon memory in realistic agentic settings. AMA-Bench combines real-world agent trajectories from representative applications with expert-curated QA, as well as synthetic trajectories that scale to arbitrary horizons with rule-based QA. Our study shows that existing memory systems underperform because they fail to capture causal and objective information and rely heavily on lossy similarity-based retrieval. We further propose **AMA-Agent**, a memory system based on causality-graph construction and tool-augmented retrieval. AMA-Agent achieves **57.22%** accuracy on AMA-Bench, outperforming the strongest baseline by **11.16%**. Resources are available at: [https://ama-bench.github.io/](https://ama-bench.github.io/).
Key insights
What problem does AMA-Bench address?
Existing memory benchmarks focus on dialogue, but real agents interact with environments through machine-generated trajectories with causal dependencies. AMA-Bench evaluates memory on these agent-environment interactions using real-world and synthetic subsets.
How does AMA-Agent differ from prior memory methods?
Unlike lossy compression or similarity-based RAG, AMA-Agent uses a Causality Graph preserving state dependencies and tool-augmented retrieval. Under Qwen3-32B on 2,496 real-world QA pairs, it averages 0.5722 accuracy, beating HippoRAG2 (0.4480) and EMem (0.4610).
What are the practical uses of these findings?
The benchmark and AMA-Agent guide memory design for agents in web navigation, software engineering, text-to-SQL, gaming, and embodied AI tasks. The paper demonstrates a prototype, not a production deployment, showing QA accuracy correlates with end-to-end task success.
What limitations do the authors acknowledge?
The authors state the study focuses on in-episode memory only. They explicitly identify extending these evaluation standards to cross-task scenarios involving lifelong learning as future work.