Cache-Augmented Generation (CAG) is an alternative paradigm to RAG that eliminates real-time retrieval entirely. Instead of searching for relevant documents at query time, CAG preloads the entire knowledge base into a long-context LLM’s context window and precomputes the model’s internal key-value (KV) cache. When a user asks a question, the model already “knows” all the documents — it simply generates an answer from its cached state with no retrieval step needed.
The concept was formalized in the December 2024 paper “Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks” by Huang et al., which demonstrated that for well-scoped knowledge domains, CAG can match or exceed RAG accuracy while reducing generation latency by 40x — from 94 seconds (RAG) to 2.3 seconds (CAG) on the HotPotQA benchmark.
CAG is made possible by two converging trends: modern LLMs now support context windows of 128k to 2M tokens (Claude supports 200k, Gemini supports 2M), and inference providers offer KV cache persistence across requests. Together, these capabilities mean you can “pre-read” a corpus once and serve thousands of queries against it without re-reading or re-searching each time. For teams with well-defined, relatively stable knowledge bases, CAG offers a dramatically simpler architecture with better latency characteristics than RAG.
CAG moves retrieval from query-time to build-time. Instead of “search, then read,” the model has already read everything. Each query simply triggers generation against pre-digested content, skipping embedding, vector search, reranking, and prompt construction entirely.
The entire relevant corpus is concatenated and fed into the LLM as a single massive prompt during a one-time preprocessing step:
# Preloading phase (runs once, offline)
documents = load_all_documents("./knowledge_base/")
concatenated = "\n---\n".join([doc.text for doc in documents])
# Construct the preloading prompt
preload_prompt = f"""You are a knowledge assistant. Below is the
complete knowledge base you will use to answer questions.
<knowledge_base>
{concatenated}
</knowledge_base>
You will now receive questions about this content. Answer based
only on the information provided above."""
# Run the prompt through the model to compute KV cache
kv_cache = model.precompute_kv_cache(preload_prompt)
save_to_disk(kv_cache, "knowledge_cache.bin")The model processes all documents through its transformer layers, computing the key and value matrices at each attention layer. These KV states encode how every token in the knowledge base relates to every other token — capturing cross-document connections that RAG’s per-chunk retrieval misses. The resulting cache is saved to disk (or kept in GPU memory) for reuse across requests.
When a user query arrives, the precomputed KV cache is loaded and the query is appended:
# Inference phase (runs per query)
kv_cache = load_from_disk("knowledge_cache.bin")
user_query = "What is our policy on remote work?"
# The model generates using the preloaded cache
# No embedding, no search, no reranking needed
response = model.generate(
prompt=user_query,
kv_cache=kv_cache,
max_tokens=500
)Because the model has already computed attention states for all knowledge base tokens during preloading, it only needs to compute attention for the new query tokens against the cached states. This dramatically reduces computation — from processing 100,000+ tokens per query (if you re-read everything) to processing just the query’s tokens (typically 10–50 tokens).
After each query, the KV cache accumulates the query and response tokens. CAG uses a reset mechanism to truncate these appended tokens, restoring the cache to its original state for the next query:
The KV cache is split into two regions: the static prefix (the knowledge base, never changes) and the dynamic suffix (query + response tokens, reset between queries). After generating a response, the dynamic suffix is truncated, returning the cache to its base state. This ensures consistent behavior across queries without recomputing the expensive prefix.
When the underlying knowledge base changes, the cache must be recomputed — the entire preloading phase runs again. This makes CAG less suitable for rapidly changing data but perfectly suited for stable reference material.
RAG Pipeline (per query):
Query → Embed → Vector Search → Rerank → Construct Prompt → LLM Generate
Components: Embedding model, Vector DB, Reranker, LLM
Latency: 1500-4000ms
CAG Pipeline (per query):
Query → Load KV Cache → LLM Generate
Components: LLM only
Latency: 50-500ms
CAG Precomputation (one-time):
All Documents → LLM Prefill → Save KV Cache
Time: Minutes to hours (depends on corpus size)
Storage: 1-50 GB (depends on model and context length)CAG is the better choice when several conditions align:
A SaaS company with 150 pages of product documentation can preload the entire corpus into a CAG system. Every customer query gets answered with full awareness of all documentation — no retrieval errors, no missed context, and sub-second response times. When documentation is updated (typically weekly or monthly), the cache is recomputed in a scheduled job.
Internal compliance systems with fixed regulatory documents (SOC 2 requirements, GDPR articles, company policies) are ideal CAG candidates. The corpus is stable, well-defined, and requires cross-reference reasoning (“Does policy X conflict with regulation Y?”) that RAG’s chunk-level retrieval struggles with.
Major API providers have enabled CAG-style patterns through caching features:
CAG is not a universal replacement for RAG. RAG remains superior when: