Prompt Compression (LLMLingua)

Overview

Prompt compression is the technique of algorithmically removing non-essential tokens from a prompt while preserving its semantic intent. The goal is to reduce the number of tokens sent to an LLM — cutting costs, reducing latency, and sometimes even improving output quality by eliminating noise. The most prominent family of algorithms in this space is Microsoft Research’s LLMLingua, which can achieve up to 20x compression ratios with minimal performance degradation.

The motivation is simple arithmetic: if your RAG pipeline retrieves 15,000 tokens of context but only 4,000 of those tokens are truly relevant to the query, you’re paying 3.75x more than necessary — and you’re diluting the model’s attention across irrelevant material. Prompt compression sits between retrieval and generation, acting as an intelligent filter that strips away filler words, redundant phrasing, and low-information-density passages.

Unlike simple truncation (which blindly cuts from the end) or summarization (which requires an expensive LLM call), prompt compression operates at the token level using small, fast models. A 335M-parameter BERT-class model can compress a prompt in milliseconds, making it practical as a pipeline stage even in latency-sensitive applications.

How It Works

The Core Idea: Perplexity-Based Token Removal

The original LLMLingua algorithm (EMNLP 2023) is built on an elegant insight: tokens that a small language model finds unsurprising (low perplexity) are likely redundant and can be safely removed. A token like “the” before “transformer architecture” adds almost no information — the model expects it — so it can be dropped. Conversely, “HNSW” in a sentence about vector databases has high perplexity (the model didn’t predict it) and must be preserved.

LLMLingua Architecture (v1)

The original LLMLingua uses a coarse-to-fine three-module pipeline:

1. Budget Controller: Allocates different compression ratios to different prompt components. Instructions receive gentle compression (2x), demonstrations get moderate compression (5x), and retrieved documents get aggressive compression (10-20x). This reflects the insight that not all parts of a prompt are equally important.
2. Iterative Token-level Compression: Within each component, tokens are scored by perplexity using a causal language model. Rather than scoring all tokens independently, LLMLingua iterates: after removing the lowest-scoring tokens in one pass, it re-evaluates remaining tokens with updated context, capturing conditional dependencies between tokens.
3. Distribution Alignment: The small compression model is fine-tuned to align with the target LLM’s token distribution. This prevents the compressor from removing tokens that the small model finds predictable but the large model relies on.

LLMLingua-2: Token Classification Approach

LLMLingua-2 (ACL 2024) fundamentally rethinks the approach. Instead of using perplexity as a proxy for importance, it directly trains a model to classify each token as “keep” or “discard”:

Key improvements in LLMLingua-2:

• Bidirectional context: Uses a BERT-style encoder that sees tokens in both directions, capturing importance signals that unidirectional models miss.
• Task-agnostic: Trained on diverse compression examples, generalizing across summarization, QA, reasoning, and conversation tasks.
• 3-6x faster: The small encoder model runs significantly faster than the LLaMA-7B model used in v1.
• Better out-of-domain generalization: The classification approach handles unseen document types more robustly than perplexity-based scoring.

LongLLMLingua: Optimized for RAG

LongLLMLingua extends the family specifically for long-context RAG scenarios. It adds a question-aware compression step: rather than compressing documents independently, it scores token importance relative to the user’s query. Tokens that are semantically related to the query are preserved even if they have low perplexity, while tokens that are predictable and query-irrelevant are aggressively pruned.

Integration in a RAG Pipeline

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    device_map="cpu"
)

# Retrieved chunks from vector search
retrieved_context = [chunk1, chunk2, chunk3, chunk4, chunk5]
full_prompt = system_prompt + "\n".join(retrieved_context) + user_query

# Compress to fit budget
compressed = compressor.compress_prompt(
    full_prompt,
    instruction=system_prompt,      # preserve instructions
    question=user_query,            # preserve the query
    target_token=4000,              # target compressed size
    context_budget="+200",          # allow slight overflow
    dynamic_context_compression_ratio=0.3  # 70% compression on context
)

# Send compressed prompt to LLM
response = llm.generate(compressed["compressed_prompt"])
# Original: 15,000 tokens → Compressed: 4,000 tokens
# Cost savings: 73% | Latency reduction: ~60%

Practical Applications

Cost Reduction at Scale

For a system handling 100,000 queries/day with an average prompt size of 12,000 tokens, even 5x compression saves significant costs. At GPT-4’s input pricing (~$10/M tokens), compression from 12K to 2.4K tokens per query saves roughly $960/day — over $28,000/month. The compression model itself runs on CPU and costs a fraction of a cent per call.

Latency-Sensitive Applications

LLM inference time scales roughly linearly with input token count. Compressing 20K tokens to 5K tokens can reduce time-to-first-token by 40-60%, which is critical for real-time chat interfaces, voice assistants, and search experiences where users expect sub-second responses.

Fitting More Signal into Fixed Windows

When using models with smaller context windows (4K-8K tokens), prompt compression enables you to fit the equivalent of 40K tokens of retrieved context into the available space. This is particularly valuable when using cost-effective smaller models that have tight context limits but sufficient reasoning capability for the task.

When NOT to Compress

Prompt compression is not appropriate for every use case:

• Code generation: Removing tokens from code examples can break syntax. Use selective compression that exempts <pre> blocks.
• Legal/medical text: Every word may be legally significant. Compression risks altering meaning in ways that create liability.
• Already-concise prompts: If your prompt is under 2,000 tokens, the overhead of compression outweighs the savings.
• Structured data: JSON, XML, and tabular data have rigid formatting where removing tokens corrupts the structure.

Key Takeaways

• Prompt compression removes non-essential tokens algorithmically, achieving up to 20x compression while preserving semantic intent.
• LLMLingua v1 uses perplexity-based scoring from a small causal LM to identify removable tokens, with a budget controller that compresses different prompt sections at different ratios.
• LLMLingua-2 reformulates compression as a binary token classification task using a BERT encoder, trained via data distillation from GPT-4 — 3-6x faster and better at out-of-domain generalization.
• LongLLMLingua adds query-aware compression for RAG scenarios, preserving query-relevant tokens regardless of their perplexity.
• Practical savings: 40-70% cost reduction, 40-60% latency improvement at 5x compression ratios.
• Avoid compressing code, structured data, and legally sensitive text — compression is best suited for natural language retrieval contexts.
• The compression model (~335M parameters) runs on CPU in milliseconds, adding negligible overhead to the pipeline.