Serving large language models in production is fundamentally different from running inference in a Jupyter notebook. Production LLM workloads demand high throughput (hundreds or thousands of requests per second), low latency (sub-second time-to-first-token), and efficient GPU memory utilization. Inference serving engines are specialized frameworks designed to solve exactly these problems by optimizing how models are loaded, how requests are batched, and how GPU memory is managed.
The three dominant inference serving engines in the ecosystem are vLLM (from UC Berkeley), Text Generation Inference (TGI) (from Hugging Face), and TensorRT-LLM (from NVIDIA). Each takes a different approach to the core challenge: getting maximum performance out of expensive GPU hardware while maintaining compatibility with a broad set of models. A fourth contender, SGLang, has also gained significant traction with its RadixAttention approach.
Understanding these engines is essential for any AI engineer building production systems. The choice of serving engine can mean the difference between needing 4 GPUs or 16 GPUs for the same workload, directly impacting infrastructure costs by 2-4x.
vLLM’s core innovation is PagedAttention, an attention algorithm inspired by virtual memory paging in operating systems. Traditional LLM serving allocates a contiguous block of GPU memory for each request’s key-value (KV) cache. Since the final output length is unknown in advance, systems must pre-allocate for the maximum possible sequence length, wasting 60-80% of KV cache memory through fragmentation.
PagedAttention solves this by partitioning each sequence’s KV cache into fixed-size blocks (typically 16 tokens, ~12.8KB for a 13B parameter model). These blocks can live anywhere in GPU memory, mapped through a block table similar to an OS page table. When a new token is generated, only a new block is allocated on demand. When a request completes, its blocks are immediately returned to the free pool.
PagedAttention treats the KV cache like virtual memory — splitting it into small, non-contiguous blocks mapped through a block table. This reduces memory waste from 60-80% down to under 4%, enabling 2-4x more concurrent requests on the same GPU.
vLLM combines PagedAttention with continuous batching, prefix caching, and speculative decoding. It delivers up to 24x higher throughput than naive HuggingFace Transformers inference and supports a broad range of model architectures out of the box.
# Starting a vLLM server
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3-70B-Instruct",
tensor_parallel_size=4,
gpu_memory_utilization=0.90)
params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain quantum computing"], params)
# Or via the OpenAI-compatible API server:
# python -m vllm.entrypoints.openai.api_server \
# --model meta-llama/Llama-3-70B-Instruct \
# --tensor-parallel-size 4TensorRT-LLM is NVIDIA’s optimized inference library that compiles LLM architectures into highly optimized CUDA kernels specifically tuned for NVIDIA GPUs. It achieves the lowest latency numbers among serving engines by leveraging custom attention kernels, FP8/FP4/INT4 quantization, in-flight batching, and deep integration with the NVIDIA software stack (CUDA, cuBLAS, cuDNN).
The key differentiator is the compilation step: models are converted into optimized TensorRT engine files that fuse operations, eliminate memory copies, and use hardware-specific instructions. This produces typical throughput of ~180-220 requests/second with 35-50ms time-to-first-token on supported models — roughly 30-50% faster than vLLM on equivalent hardware.
The tradeoff is flexibility. TensorRT-LLM requires an explicit model conversion step, supports fewer model architectures than vLLM, and is tightly coupled to NVIDIA GPUs. It’s the right choice when you’re running a specific model at massive scale on NVIDIA hardware and need every last drop of performance.
Hugging Face’s TGI prioritizes developer experience and tight integration with the Hugging Face ecosystem. TGI v3 introduced several significant improvements: chunked prefill for long prompts, prefix caching, and optimized scheduling that can process about 3x more tokens in the same GPU memory compared to v2.
TGI’s standout capability is long-context processing. On prompts exceeding 200,000 tokens, TGI v3 has demonstrated 13x speedups over vLLM for conversation replies — processing in ~2 seconds what takes vLLM ~27.5 seconds. This makes it particularly strong for RAG workloads and document-heavy applications.
TGI achieves throughput of ~100-140 requests/second with 60-90ms time-to-first-token. While these numbers trail vLLM and TensorRT-LLM in raw throughput, TGI’s strength lies in its one-line deployment of any Hugging Face model and its production-ready features like token streaming, watermarking, and built-in safety controls.
SGLang (from Stanford) introduces RadixAttention, which uses a radix tree to automatically detect and reuse shared prefixes across requests. When multiple requests share the same system prompt or few-shot examples, SGLang caches and reuses the computed KV cache for shared prefixes, avoiding redundant computation. On workloads with high prefix overlap (common in production), SGLang can match or exceed vLLM’s throughput.
The choice of inference engine depends on your specific workload characteristics:
In production, these engines are typically deployed behind a load balancer with multiple GPU workers. A common pattern is to run vLLM with tensor parallelism across 4-8 GPUs per model replica, with an autoscaler that adds replicas based on request queue depth. For example:
# Kubernetes deployment with vLLM
apiVersion: apps/v1
kind: Deployment
spec:
replicas: 2
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- "--model=meta-llama/Llama-3-70B-Instruct"
- "--tensor-parallel-size=4"
- "--max-model-len=8192"
- "--gpu-memory-utilization=0.90"
resources:
limits:
nvidia.com/gpu: 4The efficiency gains are substantial. Moving from naive HuggingFace inference to vLLM typically reduces GPU requirements by 3-5x for the same throughput. Organizations serving millions of requests per day report infrastructure savings of $50,000-$200,000/month by switching to optimized serving engines. The additional step of using TensorRT-LLM compilation can squeeze out another 30-50% on top of that for NVIDIA-only deployments.