Overview
Every neural network is, at its core, a collection of weight matrices. How those weights are represented — their numerical format, sparsity pattern, and structural organization — determines the model’s memory footprint, inference speed, and deployment options. There are four fundamental approaches to weight representation, each trading off a different axis of efficiency.
This page is a comparative map of all four approaches. It links to the detailed topic pages for each technique, provides a side-by-side comparison table, and explains how these approaches compose in production systems. If you’re deciding how to optimize a model for deployment, start here.
The four representations are: Dense (full-precision, all weights active), Sparse/MoE (only a subset of weights activated per input), Quantized (lower numerical precision), and Pruned (weights permanently removed). Understanding when and how to apply each — and how they stack — is essential for deploying large models efficiently.
The Four Weight Representations
1. Dense (Full-Precision)
The baseline: every weight is stored in full precision (FP32 or BF16/FP16) and every weight participates in every forward pass. This is how models are typically trained. A 70B-parameter model in BF16 requires ~140 GB of memory just for weights — before accounting for activations, KV cache, or optimizer state.
- How it works: Standard matrix multiplications with all weights. No special hardware or software support needed.
- Benefit: Maximum model quality. No accuracy loss from compression. Simplest to train, debug, and interpret.
- Cost: Highest memory, compute, and energy consumption. Large models may not fit on available hardware.
2. Sparse / Mixture-of-Experts (MoE)
Instead of activating all parameters for every input, Mixture-of-Experts models route each token to a small subset of specialized “expert” sub-networks (typically 2 out of 8–64 experts). The total parameter count is high (for knowledge capacity), but the active parameter count per token is much lower (for compute efficiency). See Mixture of Experts (MoE) for the full treatment.
- How it works: A learned router network selects which experts process each token. Only selected expert weights are used in the forward pass.
- Benefit: Scale model capacity (total parameters) independently of per-token compute (active parameters). Mixtral 8x7B has 47B total params but only ~13B active per token.
- Cost: All expert weights must be stored in memory (high memory footprint). Communication overhead in distributed settings. Router imbalance can waste capacity.
Models can be born as MoE or converted post-hoc via Sparse Upcycling or ToMoE.
3. Quantized (Low-Bit)
Quantization reduces the numerical precision of weights from 16-bit or 32-bit floating point to lower bit-widths — typically 8-bit, 4-bit, or even 2-bit integers or custom formats. This shrinks memory proportionally and enables faster integer arithmetic on supporting hardware. See Quantization for techniques like GPTQ, AWQ, and GGUF.
- How it works: Weights are mapped from floating-point to a discrete grid of lower-precision values. Post-training quantization (PTQ) applies this after training; quantization-aware training (QAT) incorporates it during training.
- Benefit: Linear reduction in memory (4-bit = 4× smaller than FP16). Faster inference on hardware with integer/low-bit support. Minimal accuracy loss with modern methods (GPTQ, AWQ) at 4-bit.
- Cost: Some accuracy degradation, especially below 4-bit. Outlier weights need special handling (mixed precision). Quantization artifacts can emerge in long-context or reasoning-heavy tasks.
4. Pruned (Weights Removed)
Pruning permanently removes weights deemed unimportant — setting them to zero (unstructured) or removing entire rows/columns/heads (structured). The model becomes smaller and potentially faster. See Neural Network Pruning for SparseGPT, Wanda, and structured pruning.
- How it works: An importance criterion (magnitude, Hessian-based, activation-aware) identifies which weights to zero out or remove. One-shot methods like SparseGPT do this without retraining.
- Benefit: Reduces model size and (with hardware support) inference compute. Semi-structured 2:4 sparsity achieves ~2× speedup on NVIDIA Ampere/Hopper GPUs. Structured pruning produces a smaller dense model that runs on any hardware.
- Cost: Unstructured sparsity needs sparse kernels for speedup. Accuracy degrades at high sparsity (>60%). Irreversible — pruned weights cannot be recovered without retraining.
Comparison Table
Representation
How It Works
Primary Benefit
Primary Cost
Stacks With
Dense
All weights active, full precision (FP16/BF16/FP32)
Maximum accuracy; simplest to train and debug
Highest memory and compute; may not fit on target hardware
Baseline — all others start here
Sparse / MoE
Router selects subset of expert FFN blocks per token
High capacity with low per-token compute (3–4× fewer active params)
Full model in memory; router complexity; communication overhead
Quantization, Pruning
Quantized
Weights mapped to lower bit-width (8-bit, 4-bit, 2-bit)
Linear memory reduction (4-bit = 4× smaller); faster integer ops
Accuracy loss below 4-bit; outlier handling needed; quantization artifacts
MoE, Pruning, Distillation
Pruned
Unimportant weights zeroed (unstructured) or removed (structured)
Smaller model; 2× speedup with 2:4 sparsity on NVIDIA GPUs
Sparse kernels needed for unstructured speedup; accuracy loss at >60% sparsity
Quantization, MoE
These Compose: Real-World Stacking
The four representations are not mutually exclusive — in fact, production models routinely combine multiple techniques for maximum efficiency. The key insight is that each technique targets a different source of redundancy:
- MoE reduces compute redundancy (not all knowledge is needed for every token).
- Quantization reduces precision redundancy (16-bit is more precision than needed for inference).
- Pruning reduces parameter redundancy (many weights contribute negligibly).
Key Concept: The Composition Stack A pruned, 4-bit-quantized MoE model upcycled from a dense checkpoint represents the full composition stack: start with a dense pre-trained model → upcycle into MoE (sparse activation) → prune inactive or low-importance weights → quantize to 4-bit. Each step compounds the savings. A model that would require 140 GB as a dense FP16 checkpoint might serve at under 20 GB as a pruned, quantized MoE — fitting on a single consumer GPU.
Real Examples of Composition
- Mixtral 8x7B quantized to 4-bit (GGUF/AWQ): An MoE model (47B total, ~13B active) quantized for consumer deployment. Fits in ~26 GB at Q4_K_M — a single 32 GB GPU or split across CPU+GPU.
- 2:4 sparse Llama + 4-bit quantization: Red Hat’s Sparse Llama applies NVIDIA 2:4 semi-structured sparsity to Llama models, then quantizes to 4-bit. The combination delivers 3–5× inference speedup over the dense FP16 baseline on Ampere GPUs.
- DeepSeek-V3: Uses MoE architecture (671B total, 37B active) with FP8 mixed-precision training — combining sparse activation with reduced precision from the start of training, not just at deployment.
- SqueezeLLM: A “dense-and-sparse” quantization method that stores outlier weights in a sparse format and the majority in ultra-low-bit (3–4 bit), combining quantization and sparsity at the weight-storage level.
Choosing the Right Combination
The typical decision tree for deploying a large model:
- Does it fit in memory at FP16? If yes, start with quantization (4-bit AWQ/GPTQ) for easy 4× memory savings with minimal accuracy loss.
- Need more compute efficiency? Apply 2:4 pruning before quantization for hardware-accelerated sparsity on NVIDIA GPUs.
- Need massive scale with lower per-token cost? Use or upcycle to an MoE architecture, then quantize the experts.
- Extreme constraints (edge, mobile)? Combine structured pruning + aggressive quantization (2–4 bit) + knowledge distillation from the full model.
Fine-Tuning with Low-Bit Representations
Quantization doesn’t just help inference — it enables fine-tuning of models that wouldn’t otherwise fit in GPU memory. QLoRA quantizes the base model to 4-bit NormalFloat and trains only small LoRA adapters in full precision. This allows fine-tuning a 65B model on a single 48 GB GPU — democratizing access to large model customization.
Key Takeaways
- The four weight representations — Dense, Sparse/MoE, Quantized, and Pruned — each target a different source of model redundancy.
- Dense is the baseline with maximum accuracy but highest cost. All other representations trade some accuracy for efficiency.
- MoE scales capacity independently of per-token compute. Quantization provides linear memory savings. Pruning removes unneeded weights for size and speed gains.
- These techniques compose: pruned + quantized + MoE models are common in production and can reduce memory by 10× or more vs. the dense baseline.
- Start with quantization (easiest, most universal), add pruning for hardware-supported speedup, and consider MoE for the largest scale deployments.
Deep Dive Topics