Sparse upcycling creates an MoE model by cloning FFN layers into identical experts and then training them to diverge. ToMoE (2025) takes a fundamentally different approach: it discovers that expert-like structure already exists inside a dense model’s FFN layers and extracts it — without updating any of the original model weights. The dense model’s neurons naturally cluster into functional groups; ToMoE reveals these groups and routes tokens to them selectively.
The insight builds on earlier work by Zhang et al. (2022), called MoEfication, which first showed that FFN neurons in trained Transformers can be partitioned into functional groups that behave like experts. However, MoEfication was limited to ReLU-based architectures and separated expert construction from router training into two stages, leading to suboptimal results. ToMoE unifies these steps using differentiable dynamic structural pruning, handles modern GLU-based architectures (LLaMA, Qwen, Phi), and consistently outperforms both pruning and upcycling baselines.
For AI engineers, ToMoE represents a zero-training-cost path from a dense model to an MoE model. If you need to reduce inference compute for a deployed dense model without any fine-tuning budget, ToMoE offers a compelling option — the only parameters trained are the lightweight routers, while all original model weights remain frozen.
A trained dense FFN layer has thousands of neurons (rows of W1 / columns of W2). In practice, different subsets of these neurons activate for different types of inputs. Some neurons fire strongly on code tokens, others on mathematical expressions, others on natural language narrative. ToMoE exploits this natural specialization by partitioning neurons into groups (experts) and training a router to select which groups to activate per token.
Dense models are not actually “dense” in their activation patterns — most neurons in an FFN layer contribute minimally to any given token’s output. Studies show that for a typical input token, only 10–30% of FFN neurons produce significant activations. ToMoE formalizes this observation: if most neurons are idle for most tokens, you can group the active neurons into experts and skip the rest, achieving MoE-like sparsity without architectural changes during training.
ToMoE frames the dense-to-MoE conversion as a structural pruning problem with a twist: instead of permanently removing neurons, it dynamically selects which neuron groups (experts) to activate per token. The method works in a single unified stage:
N groups (e.g., 8 or 16 experts). ToMoE uses a differentiable partitioning scheme that jointly optimizes the grouping and the routing.k expert groups to activate for each token. Only the router weights are updated — all original FFN weights are frozen.# ToMoE conceptual pipeline (simplified)
for layer in model.ffn_layers:
# Partition neurons into N expert groups
experts = partition_neurons(layer.W1, layer.W2, num_experts=8)
# Train a router (only trainable parameters)
router = nn.Linear(d_model, num_experts)
# At inference: activate top-k expert groups per token
active_experts = router.top_k(hidden_state, k=2)
output = sum(experts[i](hidden_state) for i in active_experts)MoEfication (Zhang et al. 2022) was the first to demonstrate that dense FFN layers can be decomposed into expert-like groups. It used a two-stage process:
This two-stage approach has limitations: the clustering is done without considering the router, and the router is trained without being able to influence the clustering. Additionally, MoEfication relied on ReLU activations producing exact zeros, which made sparsity patterns clear — modern architectures using SwiGLU or GeLU produce non-zero activations everywhere, breaking this assumption.
ToMoE’s key contribution is unifying expert construction and router training into a single differentiable optimization. The router learns which neuron groups to select while simultaneously influencing how neurons are grouped. This joint optimization produces significantly better results: on LLaMA-2, ToMoE at 50% sparsity outperforms MoEfication by 5–8 points on average across standard benchmarks, and it works seamlessly with modern GLU-based architectures.
ToMoE has been validated across diverse modern architectures:
ToMoE’s most compelling use case is when you need to compress a model but have no fine-tuning budget. Traditional pruning permanently removes parameters and usually requires recovery fine-tuning to restore quality. ToMoE achieves compression through dynamic sparsity — neurons are not removed, just selectively skipped — and the only training required is for the lightweight routers, which converges quickly on a small calibration dataset.
For deploying LLMs on edge devices with limited compute, ToMoE offers a path to reduce per-token FLOPs without permanent information loss. Since all original weights are preserved, the full dense model can still be activated for critical inference tasks while running in sparse mode for latency-sensitive or resource-constrained scenarios.
ToMoE provides a lens into how dense models organize their computation. By examining which neurons are grouped together and which token types activate each expert group, researchers can gain insights into the model’s internal specialization — even though no MoE training ever occurred. This can inform architecture design decisions for future models.
ToMoE-derived sparsity is orthogonal to quantization and can be combined with it for multiplicative compression. For example, a ToMoE model with 50% sparsity and 4-bit quantization would achieve roughly 8× total compression: 2× from activating half the neurons and 4× from reducing each weight from 16-bit to 4-bit. This combination is particularly powerful for deployment on consumer hardware.