LLM inference has a fundamental problem: different requests produce different numbers of output tokens. One user asks “What is 2+2?” (generates 3 tokens) while another asks “Write a detailed essay on climate change” (generates 2,000 tokens). In traditional static batching, both requests are grouped together and the entire batch must wait for the longest request to finish before accepting new work. The GPU sits mostly idle while the short request’s slot is wasted waiting for the long one.
Continuous batching (also called iteration-level scheduling or in-flight batching) solves this by operating at the granularity of individual token generation steps rather than complete requests. As soon as a request emits its end-of-sequence token, the scheduler immediately evicts it and slots a new request from the waiting queue — on the very next token step. This single optimization can improve throughput by up to 23x over naive static batching.
Understanding continuous batching is critical for AI engineers because it’s the core scheduling algorithm behind every major inference serving engine (vLLM, TGI, TensorRT-LLM, SGLang). It determines how efficiently your GPU utilization translates into served requests, directly impacting infrastructure costs and user-facing latency.
In static (or naive) batching, the inference server collects incoming requests until it has a full batch (or a timeout expires), then processes the entire batch together. Every sequence in the batch goes through the model’s forward pass simultaneously, which is efficient for GPU utilization during matrix operations. However, the critical flaw is that all sequences must complete before any new requests can enter the batch.
In static batching, if a batch contains requests generating 10, 50, 200, and 1,000 tokens, the GPU runs all four through every forward pass — but three of the four slots become “bubbles” (wasted compute) as soon as their sequences finish, while the batch waits for the 1,000-token request to complete.
Anyscale’s benchmarks show that static batching throughput drops from ~200 tokens/second to ~81 tokens/second as output length variance increases, precisely because of this bubble effect.
Continuous batching operates at the granularity of a single token generation step. The scheduler runs a loop:
Instead of waiting for an entire batch to complete, the scheduler operates at each token generation step — evicting finished sequences and admitting new ones on every iteration. This eliminates GPU bubbles and can achieve up to 23x higher throughput than static batching.
This means the batch is a living, dynamic structure. A request that arrives at time T and needs only 10 tokens can be admitted, processed, completed, and returned — all while a longer request that started earlier is still generating. The GPU never sits idle with empty batch slots as long as there are requests in the queue.
# Simplified pseudocode for a continuous batching scheduler
class ContinuousBatchScheduler:
def __init__(self, model, max_batch_size):
self.model = model
self.max_batch_size = max_batch_size
self.active_batch = [] # Currently generating sequences
self.waiting_queue = [] # Queued requests
def step(self):
# 1. Run forward pass for all active sequences
next_tokens = self.model.forward(self.active_batch)
# 2. Check for completed sequences
completed = []
for seq, token in zip(self.active_batch, next_tokens):
seq.append_token(token)
if token == EOS or seq.length >= seq.max_tokens:
completed.append(seq)
# 3. Evict completed sequences
for seq in completed:
self.active_batch.remove(seq)
seq.return_result() # Send response to client
# 4. Admit new requests from waiting queue
while (len(self.active_batch) < self.max_batch_size
and self.waiting_queue):
new_seq = self.waiting_queue.pop(0)
self.active_batch.append(new_seq)LLM inference has two distinct phases with very different computational characteristics:
Advanced schedulers handle these phases differently. Chunked prefill breaks long prompts into smaller chunks so that prefill work can be interleaved with decode steps, preventing a single long prompt from blocking decode for the entire batch. TGI v3’s chunked prefill is why it achieves 13x speedups on 200K+ token prompts — it processes prefill in chunks while continuing decode for other requests.
When the waiting queue has more requests than available batch slots, the scheduler must decide which requests to admit and which to preempt. Common policies include:
Continuous batching’s effectiveness is directly limited by GPU memory. Each active sequence requires KV cache memory proportional to its current sequence length. The scheduler must track available GPU memory and refuse to admit new requests if doing so would cause out-of-memory errors. This is where PagedAttention (covered in the Inference Serving Engines topic) becomes essential — by reducing KV cache memory waste from ~70% to ~4%, it allows the continuous batching scheduler to pack significantly more concurrent sequences into GPU memory.
The choice between batching strategies isn’t always straightforward. Continuous batching dominates for online serving (real-time chat, API endpoints) where minimizing per-request latency and maximizing concurrency are critical. However, for offline batch processing (bulk document summarization, dataset generation), static batching can actually achieve higher peak throughput at large batch sizes because it avoids the scheduling overhead of continuous batching.
Benchmarks show that at batch sizes of 64+, static batching can outperform continuous batching for offline workloads due to lower scheduling overhead. The scheduling decisions at each iteration step (checking completions, queue management, memory allocation) add overhead that matters at very high throughput.
Real-world benchmarks demonstrate the dramatic impact of continuous batching:
Key metrics to monitor in a continuous batching system:
# vLLM exposes these metrics via Prometheus
# Key metrics to monitor:
vllm:num_requests_running # Active sequences in batch
vllm:num_requests_waiting # Queued requests
vllm:gpu_cache_usage_perc # KV cache memory utilization
vllm:avg_prompt_throughput # Tokens/sec for prefill
vllm:avg_generation_throughput # Tokens/sec for decode
vllm:e2e_request_latency_seconds # End-to-end latency