# Stateful Inference for Low-Latency Multi-Agent Tool Calling Source: [Stateful Inference for Low-Latency Multi-Agent Tool Calling]() Collection index: [llms.txt]() Data updated: 2026-05-28T05:00:05.405Z Authors: Victor Norgren Published: 2026-05-25T19:27:49.000Z ## Abstract Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the $O(n\_t)$ per-turn cost of conventional serving into an $O(\\Delta\_t)$ delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is $2.1\\times$ faster per turn on a 6-turn agentic workflow and $4.2\\times$ on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Multi-turn Coding on Novel 35-turn coding workflow](): Multi-turn Coding; Novel 35-turn coding workflow; Median Time per Turn (ms): 647. - [Tool-call Latency on Novel agentic workflow 6-turn](): Tool-call Latency; Novel agentic workflow 6-turn; Median per-turn Latency (ms): 624. ## Original sources and implementation [Original paper]() [PDF]()