# Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management

Source: [Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management](<https://www.sota2.com/research/paper/69fd6e5c5497b3b670c89726>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-09T05:00:12.752Z

Authors: Haoyu Zheng, Fangcheng Fu, Jia Wu, Binhang Yuan, Yongqiang Zhang, Hao Wang, Yuanyuan Zhu, Xiao Yan, Jiawei Jiang

Published: 2026-05-07T15:57:51.000Z

## Abstract

LLM-based workflows compose specialized agents to execute complex tasks, and these agents usually share substantial context, allowing KV-Cache reuse to save computation. Existing approaches either manage KV-Cache at agent level and fail to exploit the reuse opportunities within workflows, or manage cache at the workflow level but assume that each workflow calls a static sequence of agents. However, practical workflows are typically dynamic, where the sequence of invoked agents and thus induced cache reuse opportunities depend on the context of each task. To serve such dynamic workflows efficiently, we build a system dubbed PBKV (\\textbf{P}rediction-\\textbf{B}ased \\textbf{KV}-Cache Management). For each workflow, PBKV predicts the agent invocations in several future steps by fusing the guidance from historical workflows and context of the target workflow. Based on the predictions, PBKV estimates the reuse potential of cache entries and keeps the high-potential entries in GPU memory. To be robust to prediction errors, PBKV utilizes the predictions conservatively during both cache eviction and prefetching. Experiments on three workflow benchmarks show that PBKV achieves up to $1.85\\times$ speedup over LRU on dynamic workflows, and up to $1.26\\times$ speedup over the SOTA baseline KVFlow on the static workflow.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Agentic Workflow Performance (Iterative Refinement Loops) on HoVer + LangChain](<https://www.sota2.com/research/sota/agentic-workflow-performance-iterative-refinement-loops-on-hover-plus-langchain.md>): Agentic Workflow Performance (Iterative Refinement Loops); HoVer + LangChain; Latency (s): 76.15.

- [Agentic Workflow Performance (Retry Loops) on SWE-bench + AutoGen](<https://www.sota2.com/research/sota/agentic-workflow-performance-retry-loops-on-swe-bench-plus-autogen.md>): Agentic Workflow Performance (Retry Loops); SWE-bench + AutoGen; Latency (s): 118.61.

- [Agentic Workflow Performance (Static) on FinanceBench + CrewAI](<https://www.sota2.com/research/sota/agentic-workflow-performance-static-on-financebench-plus-crewai.md>): Agentic Workflow Performance (Static); FinanceBench + CrewAI; Latency (s): 65.01.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.06472>)

[PDF](<https://arxiv.org/pdf/2605.06472.pdf>)