Neural network pruning removes the least important weights or structural units from a trained model to reduce its size and increase inference speed without materially degrading quality. The core insight is that modern deep networks are massively over-parameterized: a large fraction of their weights contribute negligibly to the final output and can be zeroed out (or entirely removed) after training.
Pruning has become especially critical in the era of large language models. A 70-billion-parameter model requires roughly 140 GB of memory in FP16 — far more than a single GPU can hold. Pruning to 50–60% sparsity can cut memory requirements nearly in half and, when combined with quantization, can bring models that once required multi-GPU clusters onto a single consumer GPU.
The field has seen a renaissance since 2023. Methods like SparseGPT (Frantar & Alistarh, ICML 2023) and Wanda (Sun et al., ICLR 2024) showed for the first time that billion-scale models can be pruned in one shot — without any retraining — while preserving nearly all of their original accuracy. These advances make pruning a practical, everyday tool for AI engineers deploying LLMs in production.
The two fundamental categories differ in what gets removed and the hardware implications:
NVIDIA’s Ampere and Hopper GPUs natively support 2:4 sparsity: exactly 2 out of every 4 contiguous weights must be zero. This pattern is structured enough for hardware acceleration (roughly 2× speedup on sparse tensor cores) but flexible enough to approach unstructured-pruning accuracy. It is the sweet spot for production deployment on NVIDIA hardware.
Every pruning method must answer: which weights are “least important”? The major approaches are:
SparseGPT (Frantar & Alistarh 2023) was the first method to prune 100B+ parameter models in a single pass. The key ideas:
W' that minimizes ‖WX − W'X‖², where X is the calibration data.# SparseGPT pseudocode (simplified)
for each layer in model:
X = collect_calibration_activations(layer, calibration_data)
H = X @ X.T # Hessian approximation
for col in columns(W):
# Find weights to prune in this column
prune_mask = select_smallest(W[:, col], target_sparsity)
# Update surviving weights to compensate
W[:, col+1:] += compensation(H, pruned_weights)On OPT-175B, SparseGPT achieves 60% unstructured sparsity with only a 0.3-point perplexity increase on WikiText2. The entire process takes under 4.5 hours on a single machine. By contrast, magnitude pruning completely collapses the model past 30% sparsity.
Wanda (Sun et al. 2024) simplifies pruning even further. It requires no gradient computation, no Hessian approximation, and no weight updates. The pruning score for each weight w_ij is:
score(w_ij) = |w_ij| × ‖X_j‖₂
# |w_ij| = magnitude of the weight
# ‖X_j‖₂ = L2 norm of the j-th input feature across calibration samples
# Prune weights with the smallest scores, per output rowThe per-output-row granularity is critical: it ensures each output neuron retains a balanced set of inputs. Despite its simplicity, Wanda matches or rivals SparseGPT across a range of models and sparsity levels, and runs significantly faster because it skips the weight compensation step.
Traditional pruning (e.g., the Lottery Ticket Hypothesis) involves cycles of prune → retrain → prune → retrain. This is impractical for LLMs where a single training run costs millions of dollars. SparseGPT and Wanda are one-shot methods: they prune a fully trained model in a single pass using only a small calibration set (no retraining). This is what makes modern LLM pruning practical.
Frankle & Carlin (2019) proposed that dense networks contain sparse subnetworks (“winning tickets”) that, when trained in isolation from the same initialization, can match the full network’s accuracy. While the original formulation is too expensive for LLMs (it requires repeated training), it provided the theoretical motivation for believing that extreme sparsity is achievable without loss of quality. SparseGPT and Wanda can be seen as practical approximations that find good sparse subnetworks post-training, without the cost of retraining.
Pruning is one of the key techniques (alongside quantization) for fitting large models onto limited hardware. A 70B model pruned to 50% sparsity and quantized to 4-bit can be served on a single 24 GB GPU. In practice, pruning and quantization are complementary — SparseGPT explicitly supports joint pruning + quantization, and the two techniques compose well because they target different aspects of redundancy.
Unstructured sparsity provides memory savings but limited speed gains without sparse kernels. Semi-structured 2:4 sparsity on NVIDIA Ampere/Hopper GPUs delivers roughly 2× throughput on matrix multiplications via sparse tensor cores. Structured pruning (removing entire heads or layers) gives the most straightforward speedups since the result is simply a smaller dense model.
In a typical production pipeline, pruning fits between training and deployment:
Structured pruning can serve as a form of architecture search: start with a large model, prune different components at different rates based on their importance, and the resulting architecture reflects the optimal allocation of capacity. This is used in practice to create efficient model variants for edge deployment.