Prompt Cache: Modular Attention Reuse for Low-Latency Inference
About
We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have overlapping text segments, such as system messages, prompt templates, and documents provided for context. Our key insight is that by precomputing and storing the attention states of these frequently occurring text segments on the inference server, we can efficiently reuse them when these segments appear in user prompts. Prompt Cache employs a schema to explicitly define such reusable text segments, called prompt modules. The schema ensures positional accuracy during attention state reuse and provides users with an interface to access cached states in their prompt. Using a prototype implementation, we evaluate Prompt Cache across several LLMs. We show that Prompt Cache significantly reduce latency in time-to-first-token, especially for longer prompts such as document-based question answering and recommendations. The improvements range from 8x for GPU-based inference to 60x for CPU-based inference, all while maintaining output accuracy and without the need for model parameter modifications.
Key insights
What problem does Prompt Cache solve?
It accelerates LLM inference by precomputing and reusing key-value attention states of repeated text segments across prompts, reducing time-to-first-token without model changes.
How does it differ from prior KV Cache work?
Unlike KV Cache or prefix sharing, it uses a Prompt Markup Language schema for modular, position-aware reuse across arbitrary prompt segments. On LongBench it cut TTFT 5-10x on GPU and 20-70x on CPU.
What practical uses does it target?
Demonstrated prototype use cases include multi-file code generation, personalized recommendations, and parameterized trip planning, all showing reduced latency with comparable output quality.
What limitations do the authors note?
It only reduces first-token latency, not subsequent generation. Memory overhead scales with cached tokens, and semantically dependent modules may degrade quality without scaffolding. Cache replacement and compression are left to future work.