Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
About
Large language models (LLMs) have triggered a new stream of research focusing on compressing the context length to reduce the computational cost while ensuring the retention of helpful information for LLMs to answer the given question. Token-based removal methods are one of the most prominent approaches in this direction, but risk losing the semantics of the context caused by intermediate token removal, especially under high compression ratios, while also facing challenges in computational efficiency. In this work, we propose context-aware prompt compression (CPC), a sentence-level prompt compression technique where its key innovation is a novel context-aware sentence encoder that provides a relevance score for each sentence for a given question. To train this encoder, we generate a new dataset consisting of questions, positives, and negative pairs where positives are sentences relevant to the question, while negatives are irrelevant context sentences. We train the encoder in a contrastive setup to learn context-aware sentence representations. Our method considerably outperforms prior works on prompt compression on benchmark datasets and is up to 10.93x faster at inference compared to the best token-level compression method. We also find better improvement for shorter length constraints in most benchmarks, showing the effectiveness of our proposed solution in the compression of relevant information in a shorter context. Finally, we release the code and the dataset for quick reproducibility and further development: https://github.com/Workday/cpc.
Key insights
What prompt problem does CPC address?
CPC reduces long LLM prompts by removing whole sentences irrelevant to a given question using a contrastively trained context-aware sentence encoder that ranks sentence relevance by embedding similarity, preserving readability unlike token-level methods.
How does CPC differ from prior compression work?
Unlike LLMLingua-family token removal that breaks sentence semantics, CPC operates at sentence level. On LongBench at 2,000 tokens, CPC scores 50.0 average versus LongLLMLingua's 48.8, while being 10.93× faster in median latency.
What are the practical uses of CPC?
CPC is evaluated as a benchmark prototype on LongBench and ZeroSCROLLS plus domain-specific datasets including PubMed, MeetingBank, and SummScreen, showing gains across QA and summarization tasks.
What limitations do the authors note?
The authors do not list explicit failure modes. Evaluation is bounded to LongBench and ZeroSCROLLS at 2,000 and 3,000 token constraints on a single A100 GPU. LLMLingua-2 remains slightly faster at inference, and performance drops when ablated loss components or dataset filters are removed.