# KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows

Source: [KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows](<https://www.sota2.com/research/paper/68fa79ed1e209131cea80099>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:18.890Z

Authors: Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, Yufei Ding

Published: 2025-07-10T03:39:23.000Z

## Abstract

Large language model (LLM) based agentic workflows have become a popular paradigm for coordinating multiple specialized agents to solve complex tasks. To improve serving efficiency, existing LLM systems employ prefix caching to reuse key-value (KV) tensors corresponding to agents' fixed prompts, thereby avoiding redundant computation across repeated invocations. However, current systems typically evict KV caches using a Least Recently Used (LRU) policy, which fails to anticipate future agent usage and often discards KV caches shortly before their reuse. This leads to frequent cache misses and substantial recomputation or swapping overhead. We present KVFlow, a workflow-aware KV cache management framework tailored for agentic workloads. KVFlow abstracts the agent execution schedule as an Agent Step Graph and assigns each agent a steps-to-execution value that estimates its temporal proximity to future activation. These values guide a fine-grained eviction policy at the KV node level, allowing KVFlow to preserve entries likely to be reused and efficiently manage shared prefixes in tree-structured caches. Moreover, KVFlow introduces a fully overlapped KV prefetching mechanism, which proactively loads required tensors from CPU to GPU in background threads for agents scheduled in the next step, thereby avoiding cache miss stalls during generation. Compared to SGLang with hierarchical radix cache, KVFlow achieves up to 1.83$\\times$ speedup for single workflows with large prompts, and up to 2.19$\\times$ speedup for scenarios with many concurrent workflows.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Agentic Workflow Performance (Static) on FinanceBench + CrewAI](<https://www.sota2.com/research/sota/agentic-workflow-performance-static-on-financebench-plus-crewai.md>): Agentic Workflow Performance (Static); FinanceBench + CrewAI; Latency (s): 80.13.

- [LLM Workflow Orchestration on Representative Workflows MMLU TAT-QA Amazon](<https://www.sota2.com/research/sota/llm-workflow-orchestration-on-representative-workflows-mmlu-tat-qa-amazon.md>): LLM Workflow Orchestration; Representative Workflows MMLU TAT-QA Amazon; Avg Relative Latency (vs Helium): 1.32.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.07400>)

[PDF](<https://arxiv.org/pdf/2507.07400.pdf>)