Choosing the right GPU for LLM inference is one of the most consequential infrastructure decisions an AI engineering team makes. The wrong choice wastes money on underutilized hardware or, worse, creates bottlenecks that throttle your entire serving pipeline. Unlike traditional compute workloads where CPU cores and clock speed dominate, LLM inference is fundamentally constrained by two GPU resources: VRAM capacity (how much model data fits on the card) and memory bandwidth (how fast the GPU can read that data).
VRAM budgeting is the discipline of calculating exactly how much GPU memory your model, its KV cache, and the inference framework overhead will consume under realistic production conditions. Getting this calculation right determines whether your model runs at all, whether it can handle your target concurrency, and whether you need one GPU or eight.
This topic covers the core formulas for estimating VRAM requirements, the key GPU specifications that matter for LLM workloads, and a practical framework for selecting hardware that matches your model size, latency targets, and budget constraints.
At its core, VRAM consumption for LLM inference breaks down into three components: model weights, the KV cache, and framework overhead. Understanding each is essential for accurate capacity planning.
Total VRAM = Model Weights + KV Cache + Framework Overhead. As a rule of thumb: VRAM ≈ (Parameters × Bytes_per_Param) + KV_Cache + 20% overhead
Model weights are the dominant consumer. At FP16 (16-bit floating point), each parameter requires 2 bytes. A 7B parameter model therefore needs ~14 GB just for weights, a 13B model ~26 GB, and a 70B model ~140 GB. Quantization reduces this proportionally: INT8 cuts it in half (1 byte per parameter), and INT4 quarters it (0.5 bytes per parameter).
# VRAM for model weights at different precisions
Model Size FP32 FP16 INT8 INT4 (Q4)
─────────────────────────────────────────────────
7B 28 GB 14 GB 7 GB 3.5 GB
13B 52 GB 26 GB 13 GB 6.5 GB
34B 136 GB 68 GB 34 GB 17.0 GB
70B 280 GB 140 GB 70 GB 35.0 GB
120B 480 GB 240 GB 120 GB 60.0 GBKV cache stores the key and value tensors for all previously generated tokens, enabling the model to attend to prior context without recomputation. KV cache size grows linearly with sequence length and batch size. For a typical 7B model serving a single request with a 4K context window, the KV cache might consume 1-2 GB. But at 128K context with 32 concurrent requests, it can balloon to tens of gigabytes — sometimes exceeding the weight memory itself.
KV_Cache = 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_element. The factor of 2 accounts for both key and value tensors stored per layer.
Framework overhead includes CUDA context, memory allocator fragmentation, intermediate activation buffers, and framework-specific data structures. This typically adds 10-20% on top of the theoretical minimum. Frameworks like vLLM use PagedAttention to manage KV cache more efficiently, reducing fragmentation waste.
VRAM capacity determines whether a model fits, but memory bandwidth determines how fast it runs. During autoregressive decoding, each token generation requires reading the entire model weights from VRAM. The throughput ceiling is therefore:
Max tokens/sec ≈ Memory_Bandwidth / Model_Size_in_Bytes. An H100 with 3.35 TB/s bandwidth serving a 14 GB FP16 7B model can theoretically decode ~239 tokens/sec per request — in practice, achieving 60-80% of this theoretical maximum.
This is why the RTX 3090 (936 GB/s) and RTX 4090 (1,008 GB/s) achieve similar inference speeds despite the 4090 having more CUDA cores — bandwidth, not compute, is the limiting factor for single-request inference. At higher batch sizes, compute becomes more relevant as operations become more arithmetic-intensive.
Consumer GPUs offer the best price-per-VRAM for small-scale or development workloads:
Data center GPUs are designed for production serving with higher bandwidth, larger VRAM, and multi-GPU interconnects:
GPU selection follows a cascading set of decisions:
# Quick decision guide
Model ≤ 7B → Single RTX 4090/5090 (dev) or L4/A10G (cloud)
Model ≤ 34B → Single H100 80GB or 2× RTX 4090 (tensor parallel)
Model ≤ 70B → Single H200/B200 or 2-4× H100 (tensor parallel)
Model > 70B → 4-8× H100/H200 or 2-4× B200 (tensor parallel)A real-world capacity planning exercise for deploying Llama 3.1 8B for a customer-facing chatbot might look like this: The model weights at FP16 require ~16 GB. Target concurrency is 50 simultaneous users with 4K context. KV cache at peak: ~8 GB. Framework overhead: ~3 GB. Total: ~27 GB. Decision: A single RTX 5090 (32 GB) handles development, but for production reliability with headroom, an H100 (80 GB) allows scaling to higher concurrency without GPU swaps.
When a model doesn’t fit at FP16, quantization is often the first lever to pull. Running a 70B model at Q4 (4-bit quantization) reduces weight memory from 140 GB to ~35 GB, making it feasible on a single 80 GB H100 with room for KV cache. Modern quantization methods like GPTQ and AWQ maintain 95-98% of FP16 quality for most tasks, making this an extremely practical optimization.
In cloud deployments, the cost calculus changes because you’re renting by the hour. An H100 at $3/hour serving 1,000 tokens/sec costs ~$0.003 per 1K tokens. Two A100s achieving the same throughput at $2/hour each cost $0.004 per 1K tokens. Sometimes the more expensive GPU is the cheaper option — always compute cost-per-token, not just hourly rate.
Theoretical VRAM calculations provide a starting point, but production decisions should always be validated with benchmarks. Use tools like vllm bench or llmperf to measure actual throughput, latency percentiles (P50, P95, P99), and VRAM utilization under realistic traffic patterns on your candidate GPUs. The gap between theoretical and measured performance can be 20-40%.