The key-value (KV) cache is the single largest memory consumer during LLM inference, often exceeding the memory footprint of the model weights themselves for long sequences. During autoregressive text generation, each new token requires attending to all previously generated tokens. Without a cache, the model would redundantly recompute the key and value projections for every prior token at each generation step, making inference prohibitively slow.
The KV cache solves this by storing the computed key and value tensors from all attention layers so they can be reused. However, naive KV cache management leads to severe memory fragmentation—traditional systems waste 60–80% of allocated KV cache memory through internal and external fragmentation. This waste directly limits throughput: fewer concurrent sequences can fit in GPU memory, leaving expensive hardware underutilized.
Modern KV cache optimization techniques, led by PagedAttention and its successors, have reduced this waste to under 4%, enabling 2–4x throughput improvements on the same hardware. Understanding these techniques is essential for anyone deploying LLMs at production scale, where memory efficiency directly translates to cost savings and serving capacity.
In a transformer’s attention mechanism, each token in the input sequence is projected into three vectors: query (Q), key (K), and value (V). During generation, the model produces one token at a time. For each new token, its query vector must attend to the key and value vectors of all preceding tokens. Rather than recomputing K and V for the entire history, the KV cache stores these tensors and appends new entries as tokens are generated.
For a single sequence: cache_size = 2 × num_layers × num_heads × head_dim × seq_len × bytes_per_element. For a 13B-parameter model with FP16 precision and a 2048-token sequence, this works out to roughly 800MB per sequence—meaning just a handful of concurrent requests can exhaust a GPU’s entire memory.
Traditional serving systems allocate a single contiguous memory block for each sequence’s KV cache, sized for the maximum possible sequence length. This creates two forms of waste:
Together, these fragmentation sources cause 60–80% of KV cache memory to be wasted in conventional systems, severely limiting the number of sequences that can be served concurrently.
PagedAttention, introduced by the vLLM project, applies operating system virtual memory concepts to KV cache management. Instead of allocating one large contiguous buffer per sequence, PagedAttention divides the KV cache into fixed-size blocks (typically 16 tokens per block) that can be stored anywhere in GPU memory.
Think of blocks as memory pages, tokens as bytes, and sequences as processes. Just as an OS maps contiguous virtual addresses to scattered physical pages, PagedAttention maps a sequence’s logically contiguous KV entries to physically non-contiguous blocks via a block table.
The block table is a per-sequence data structure that maps logical block indices to physical block addresses in GPU memory. A global free-block pool manages allocation: when a sequence needs more cache space, it grabs a block from the pool. When a sequence completes, its blocks return to the pool for immediate reuse.
# Conceptual block table for a sequence of 40 tokens (block_size=16)
# Logical Block 0 (tokens 0-15) -> Physical Block 7
# Logical Block 1 (tokens 16-31) -> Physical Block 2
# Logical Block 2 (tokens 32-39) -> Physical Block 14 (partially filled)
block_table = {
"sequence_42": [7, 2, 14] # Maps logical -> physical block indices
}
# Attention kernel reads K,V from scattered physical blocks
# using the block table for indirectionMemory waste now only occurs in the last block of each sequence (which may be partially filled), reducing total waste to under 4%. This near-optimal utilization allows vLLM to serve 2–4x more concurrent sequences than systems using contiguous allocation.
When generating multiple completions for the same prompt (e.g., beam search or parallel sampling), all candidates share the same prompt KV cache. PagedAttention implements copy-on-write (CoW): shared blocks are referenced by multiple sequences, and only when a sequence diverges (generates a different token) is the affected block duplicated. This reduces memory overhead for parallel sampling by up to 55%, yielding up to 2.2x throughput improvement for beam search workloads.
Many production workloads share common prompt prefixes—system prompts, few-shot examples, or RAG context templates. Prefix caching detects when a new request’s prompt begins with the same tokens as a cached prefix, allowing the KV cache for those tokens to be reused without recomputation. vLLM’s automatic prefix caching uses a hash-based lookup: each block is identified by the hash of its token content, enabling instant prefix matching across requests.
When GPU memory is exhausted, the system must decide which cached data to evict. Common strategies include:
For very long sequences or memory-constrained environments, KV cache blocks can be offloaded from GPU memory to CPU RAM or even NVMe storage. NVIDIA reports that KV cache offloading can deliver up to 14x faster time-to-first-token (TTFT) for large input sequences compared to recomputing the KV cache from scratch. Modern systems like LMCache implement multi-tier storage hierarchies—hot blocks stay on GPU, warm blocks move to CPU, and cold blocks go to SSD—with asynchronous prefetching to hide transfer latency.
Reducing the numerical precision of cached key-value tensors is another effective optimization. While model weights are commonly quantized to INT8 or INT4, KV cache quantization requires more care because attention computations are sensitive to precision loss. Techniques like KV cache quantization to FP8 can halve cache memory requirements with minimal quality degradation, effectively doubling the number of concurrent sequences the system can handle.
Production LLM APIs (like those powering chatbots, coding assistants, and content generation services) must handle thousands of concurrent requests. PagedAttention-based systems like vLLM, TensorRT-LLM, and SGLang use these KV cache management techniques to maximize the number of concurrent sequences per GPU. A typical deployment might serve 3–5x more requests per second compared to naive implementations, directly reducing the cost per token served.
Applications like document analysis, code review, and multi-turn conversations with long histories push KV cache sizes to tens of gigabytes per sequence. KV cache offloading and eviction strategies make it feasible to serve 128K+ token contexts on standard GPU hardware without running out of memory. Without these techniques, long-context serving would require impractically large GPU clusters.
Enterprise deployments often share infrastructure across multiple applications, each with different system prompts and few-shot examples. Prefix caching ensures that the shared prompt components are computed once and reused across all requests, regardless of which tenant they belong to. LMCache reports up to 15x throughput improvement when prefix sharing is heavily utilized across tenants.
For organizations processing millions of tokens daily, KV cache optimization directly impacts infrastructure costs. By fitting more concurrent sequences per GPU, organizations can reduce the number of GPUs needed by 50–75% for the same throughput target. Combined with quantized KV caches and intelligent eviction, these techniques compound to deliver substantial infrastructure savings.