In production LLM deployments, a striking amount of compute is wasted on redundant work. Research shows that 31% of LLM queries exhibit semantic similarity to previous requests, and in many enterprise applications, every single request shares the same system prompt, few-shot examples, or RAG context template. Without caching, the model re-processes these identical prompt prefixes from scratch on every call, burning tokens and GPU cycles needlessly.
Prompt caching at the infrastructure level addresses this by storing the computational state—specifically, the KV cache tensors from the attention layers—so that the model can skip the prefill computation for repeated prompt prefixes. This is distinct from application-level response caching; prompt caching operates at the model inference layer, reducing the work the GPU must do rather than simply returning a stored response.
The impact is substantial: provider-level prompt caching can reduce input token costs by up to 90% and latency by up to 85% for qualifying requests. For high-volume production workloads where system prompts, tool definitions, and context templates are repeated across thousands of requests per minute, prompt caching is one of the highest-leverage cost optimization techniques available.
LLM inference has two phases: prefill (processing all input tokens to build the KV cache) and decode (generating output tokens one at a time). For long prompts, prefill dominates latency and compute. A 10,000-token system prompt requires computing attention across all layers for every token before the first output token can be generated. If this same system prompt appears in every request, the model performs identical matrix multiplications thousands of times per minute.
Prompt caching stores the KV tensors produced during the prefill phase. When a subsequent request begins with the same token sequence, the cached tensors are loaded directly into GPU memory, skipping prefill entirely for the cached portion. The model only needs to compute the prefill for the new, non-cached tokens.
Major LLM API providers have built prompt caching directly into their serving infrastructure. The mechanism works by matching the exact token prefix of incoming requests against recently cached prefixes:
Anthropic’s Claude API supports both automatic and explicit prompt caching:
# Automatic caching - system applies cache to last cacheable block
response = client.messages.create(
model="claude-sonnet-4-6-20250514",
max_tokens=1024,
cache_control={"type": "ephemeral"}, # Enable auto-caching
system="You are a helpful assistant with extensive knowledge...",
messages=[{"role": "user", "content": "What is prompt caching?"}]
)
# Explicit caching - fine-grained control over cache breakpoints
response = client.messages.create(
model="claude-sonnet-4-6-20250514",
max_tokens=1024,
system=[
{
"type": "text",
"text": "You are an expert assistant... [long system prompt]",
"cache_control": {"type": "ephemeral"} # Cache this block
}
],
messages=[{"role": "user", "content": "Explain transformers."}]
)Cache entries have a default 5-minute TTL that refreshes on each hit, with an optional 1-hour TTL available at higher cost. The pricing model incentivizes caching: cache writes cost 1.25x the base input token rate, but cache hits cost only 0.1x—a 90% discount. For a 10,000-token system prompt used in 1,000 requests, the savings are dramatic: instead of paying for 10 million input tokens at full price, you pay for one write (12,500 equivalent tokens) plus 999 cache hits (999,000 equivalent tokens at 10% cost), saving roughly 90% on those tokens.
OpenAI enables prompt caching by default on all API requests. Their system automatically detects shared prefixes across requests and caches them transparently. Cached input tokens are billed at 50% of the standard input token price. The minimum cacheable prefix is 1,024 tokens, and caching is applied in 128-token increments beyond that.
Production systems typically implement caching at multiple layers, each targeting a different source of efficiency:
Layer 1 – Exact Response Cache: Returns a stored response for identical queries (100% savings, but low hit rate).
Layer 2 – Semantic Cache: Uses embedding similarity to match queries that are semantically equivalent but worded differently (100% savings on hits, moderate hit rate).
Layer 3 – Prefix Cache: Reuses KV tensors for shared prompt prefixes, reducing prefill compute (50–90% savings on input tokens, very high hit rate for structured prompts).
A well-designed request flows through these layers sequentially: an exact match returns immediately, a semantic match returns a previously computed response, and failing both, prefix caching still reduces the computational cost of generating a new response.
Unlike prefix caching (which requires exact token matches), semantic caching uses embedding models to detect when a new query is functionally equivalent to a previous one. The incoming query is embedded, compared against a vector store of previous queries using cosine similarity, and if a match exceeds a confidence threshold, the stored response is returned without calling the LLM at all.
# Semantic caching with Redis
import redis
from openai import OpenAI
r = redis.Redis()
client = OpenAI()
def get_embedding(text):
return client.embeddings.create(
input=text, model="text-embedding-3-small"
).data[0].embedding
def cached_completion(query, threshold=0.95):
query_embedding = get_embedding(query)
# Search for semantically similar cached queries
results = r.ft("cache_idx").search(
Query(f"*=>[KNN 1 @embedding $vec AS score]")
.return_fields("response", "score")
.dialect(2),
query_params={"vec": np.array(query_embedding).tobytes()}
)
if results.docs and float(results.docs[0].score) > threshold:
return results.docs[0].response # Cache hit - no LLM call
# Cache miss - call LLM and store result
response = client.chat.completions.create(...)
# Store query embedding + response in Redis
return responseCache invalidation is critical for maintaining response quality. Strategies include:
Enterprise chatbots typically have long system prompts (brand guidelines, tool definitions, knowledge base context) that are identical across all user conversations. Prefix caching ensures this shared context is computed once and reused, reducing per-conversation costs by 80–90%. For a deployment handling 100,000 conversations per day with a 5,000-token system prompt, this can save thousands of dollars daily.
RAG systems retrieve and inject context documents into prompts. When multiple users ask about the same topic, the retrieved documents are often identical or highly similar. Semantic caching catches exact-match queries, while prefix caching reduces the cost of processing shared document contexts. Combined, these techniques can reduce RAG pipeline costs by 40–80%.
AI agents that make multiple LLM calls in a loop (tool use, planning, reflection) accumulate massive token counts. Each iteration typically resends the entire conversation history plus system prompt. Prompt caching ensures that the growing prefix is reused across iterations, preventing the quadratic cost explosion that would otherwise occur. Research shows that well-structured agentic prompts can maintain cache hit rates above 90% across tool-use loops.
When processing large datasets (e.g., classifying thousands of documents, extracting entities from a corpus), every item in the batch shares the same system prompt and few-shot examples. Prefix caching means the shared prompt is computed once for the entire batch, making per-item costs nearly proportional to only the unique content. This can reduce batch processing costs by an order of magnitude.