Mixture of Experts (MoE) is a neural network architecture that divides a model into multiple specialized sub-networks called “experts,” with only a subset activated for any given input. This sparse activation pattern allows models to scale to trillions of parameters while keeping the per-token computation cost comparable to a much smaller dense model.
MoE has rapidly become the dominant architecture behind state-of-the-art language models. Models like DeepSeek-V3, GPT-4, and Mixtral all use MoE to achieve massive capacity without proportionally massive inference costs. The core insight is simple: not every token needs every parameter. By routing each token through only the most relevant experts, MoE models deliver dense-model quality at a fraction of the compute budget.
For AI engineers, understanding MoE is essential because it fundamentally changes how you think about model sizing, deployment, and cost. A 1.8-trillion-parameter MoE model might only activate 100 billion parameters per token, meaning its latency and GPU requirements are closer to a 100B dense model than a 1.8T one.
In a standard Transformer, each block contains a self-attention layer followed by a feed-forward network (FFN). In an MoE Transformer, the FFN is replaced with an MoE layer that contains multiple parallel FFNs (the “experts”) and a gating network (the “router”). Each expert has the same architecture as the original FFN but learns to specialize in different types of inputs.
Unlike dense models where every parameter participates in every computation, MoE models activate only k out of N total experts per token. For example, Mixtral 8x7B has 8 experts but routes each token to only 2. The total parameter count is ~47B, but the active parameters per token are ~13B, giving dense-model-equivalent performance at a fraction of the compute cost.
The router is a small neural network — typically a single linear layer — that takes a token’s hidden state as input and produces a probability distribution over all experts. For each token, the router computes:
# Simplified router logic
router_logits = linear_layer(token_hidden_state) # Shape: [num_experts]
router_probs = softmax(router_logits) # Probability per expert
top_k_experts = top_k(router_probs, k=2) # Select top-k experts
# Compute weighted combination of expert outputs
output = sum(weight_i * expert_i(token) for weight_i, expert_i in top_k_experts)The router weights are renormalized after top-k selection so they sum to 1, and the final MoE layer output is the weighted combination of the selected experts’ outputs. The router is trained end-to-end with the rest of the model — it learns which experts are best for which inputs without explicit supervision.
Through training, experts naturally develop specializations. Some experts may become proficient at handling code tokens, others at mathematical reasoning, and still others at natural language understanding. This specialization emerges from a positive feedback loop: when an expert performs well on certain inputs, the router learns to send similar inputs to that expert more often, which further strengthens its specialization.
Importantly, this specialization happens without any explicit labeling of what each expert “should” handle. The router discovers useful divisions of labor purely through end-to-end gradient descent optimization.
A critical challenge in MoE training is expert collapse — when the router converges to always selecting the same few experts while others go unused. This is self-reinforcing: favored experts receive more training signal, become better, and get selected even more.
To combat this, MoE models use an auxiliary load-balancing loss that penalizes uneven expert utilization. This loss is added to the main training objective and encourages the router to distribute tokens more evenly across experts. The balance between routing quality and load uniformity is controlled by a hyperparameter typically set between 0.01 and 0.1.
An alternative to token-chooses-expert routing is expert choice routing, where each expert selects its top-k tokens from the batch rather than each token selecting its experts. This guarantees perfect load balancing by design, allows variable numbers of experts per token (important tokens may be processed by many experts), and has shown substantial gains in training efficiency.
MoE models offer compelling advantages but come with real engineering challenges:
MoE architecture powers many of the most capable models deployed in production today:
MoE is particularly well-suited for:
Deploying MoE models requires careful infrastructure planning. Expert parallelism distributes experts across GPUs — for an 8-expert model, you might place 2 experts per GPU across 4 GPUs. Frameworks like vLLM and TensorRT-LLM now include native MoE support with optimized all-to-all communication kernels. Memory requirements are dominated by the total parameter count (all experts loaded), so plan GPU memory budgets accordingly.