# CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference Source: [CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference]() Collection index: [llms.txt]() Data updated: 2026-06-20T05:00:05.764Z Authors: Kaizhen Tan, Rong Gu, Mingyuan Li Published: 2026-06-18T00:38:46.000Z ## Abstract Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method keeps a prefix tree over recently served evidence sequences and uses a greedy walk to place the most reusable prefix first, while leaving the serving engine and retrieved evidence set unchanged. Across three vLLM configurations, the method lowers median time-to-first-token (TTFT) by about 20-33 percent relative to retrieval-order prefix caching, without hurting answer quality in our QA tests. The greedy policy reaches 97.5 percent of the median TTFT gain from oracle ordering, indicating that most reusable prefix locality can be recovered by a simple scheduling layer between retrieval and inference. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Time-to-First-Token (TTFT) Latency on Controlled synthetic corpus Config A 1.0 (test)](): Time To First Token (TTFT) Latency; Controlled synthetic corpus Config A 1.0 (test); TTFT p50 Latency: 57.9. - [Time-to-First-Token (TTFT) Latency on Controlled synthetic corpus Config B 1.0 (test)](): Time To First Token (TTFT) Latency; Controlled synthetic corpus Config B 1.0 (test); TTFT p50: 40.1. - [Question Answering on HotpotQA full-wiki first 200 examples (val)](): Question Answering; HotpotQA full-wiki first 200 examples (val); Mean Jaccard: 0. - [Question Answering on NQ-Open first 200 examples public (val)](): Question Answering; NQ-Open first 200 examples public (val); Mean Jaccard: 6.8. - [Question Answering on TriviaQA rc.wikipedia first 200 examples (val)](): Question Answering; TriviaQA rc.wikipedia first 200 examples (val); Mean Jaccard: 0. ## Original sources and implementation [Original paper]() [PDF]()