Training a large Mixture-of-Experts (MoE) model from scratch is enormously expensive. Sparse upcycling offers an alternative: take an already-trained dense model, clone its FFN layers into multiple experts, add a randomly initialized router, and continue training for a fraction of the original cost. The result is an MoE model that outperforms both the original dense model and an MoE trained from scratch with the same total compute budget.
The technique was introduced by Komatsuzaki et al. (2022) at Google, who demonstrated it on T5 language models and Vision Transformers. Upcycled T5 models outperformed their dense counterparts on SuperGLUE by 1.5–2 points while using only about 50% additional compute beyond the original dense pretraining cost. Since then, upcycling has been adopted across the industry — NVIDIA used it to convert Nemotron-4 15B into a competitive MoE, and open-source projects like LLaMA-MoE apply the same principle to Meta’s LLaMA models.
For AI engineers, sparse upcycling is a critical technique because it lets you leverage the enormous investment already sunk into dense model training. Instead of discarding a dense checkpoint and starting over with an MoE architecture, you recycle it — preserving learned representations while gaining the inference efficiency and capacity advantages of sparse expert routing.
Not every Transformer block needs to become an MoE layer. In practice, you choose which FFN layers to convert — common configurations include converting every layer, every other layer, or just the layers in the middle and late portions of the network. The attention layers remain unchanged throughout.
# Example: Converting every-other FFN layer in a 32-layer model
for layer_idx in range(32):
if layer_idx % 2 == 0:
model.layers[layer_idx].ffn = convert_to_moe(
dense_ffn=model.layers[layer_idx].ffn,
num_experts=8,
top_k=2
)For each selected layer, the dense FFN’s weight matrices are duplicated N times to create N identical experts. If the original FFN has weights W1 and W2 (or W_gate, W_up, W2 for SwiGLU), each expert starts as an exact copy:
# Vanilla upcycling: clone dense weights into N experts
def convert_to_moe(dense_ffn, num_experts, top_k):
experts = [copy.deepcopy(dense_ffn) for _ in range(num_experts)]
router = nn.Linear(d_model, num_experts) # randomly initialized
nn.init.zeros_(router.bias)
return MoELayer(experts=experts, router=router, top_k=top_k)At initialization, every expert produces the same output for any input (since they are clones). With uniform router weights, the MoE layer’s output is identical to the original dense FFN’s output. This means the upcycled model starts from the same loss as the dense checkpoint — there is no catastrophic quality drop at the moment of conversion. Continued training then differentiates the experts.
The router (gating network) is a lightweight linear layer that maps each token’s hidden state to a probability distribution over experts. It is randomly initialized and trained alongside the rest of the model. During continued training, the router learns to specialize — sending different types of tokens to different experts based on the input patterns they carry.
More sophisticated initialization schemes exist. NVIDIA’s virtual group initialization assigns each expert a distinct representation in the router’s top-k space at startup, accelerating early specialization. The key constraint is that any initialization should not cause the model to diverge from the dense baseline at step zero.
The upcycled model is then trained on additional data with standard MoE training practices: a load-balancing auxiliary loss to prevent expert collapse, and typically a lower learning rate than the original pretraining (since the model is already well-initialized). The amount of continued training is usually 10–50% of the original pretraining compute:
The main weakness of vanilla upcycling is that all experts start identical, so early training suffers from slow differentiation — the router has no reason to prefer one clone over another. Several techniques address this:
Drop-Upcycling (2025) randomly re-initializes a fraction of each expert’s FFN weights after cloning. For example, resetting 10–20% of weights per expert introduces diversity from the start while preserving most of the pretrained knowledge. This breaks symmetry immediately and accelerates expert specialization, leading to faster convergence and better final quality than vanilla cloning.
Other approaches include adding small random noise to each expert’s weights, using different dropout masks per expert during early training, or applying delta decomposition where each expert is represented as the shared base plus a trainable low-rank or sparse difference matrix.
Upcycling is the right choice when:
Training an MoE from scratch gives you full control over expert count, routing strategy, and architecture. But it requires the full pretraining budget. Upcycling trades some architectural flexibility for a massive reduction in training cost — typically 3–10× cheaper total compute. The quality gap between upcycled and from-scratch MoE models narrows as the continued training budget increases.