Mixture of Experts — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

1. What Is Mixture of Experts?
2. The MoE Block
3. Routing Strategies
4. Load Balancing
5. Expert Parallelism
6. Major MoE Architectures (Open Frontier)
7. MoE in Vision and Multimodal
8. Training MoE Models
9. Inference for MoE
10. Frameworks and Software
11. Specialized MoE Variants
12. Expert Specialization and Analysis
13. MoE Scaling Laws
14. Production Patterns and Trade-offs
15. Common Failure Modes and Mitigations
16. Recent Frontier (2025–2026)
17. Production Stack 2026

Appendix A: Twenty-Five Things to Know

Appendix B: Decision Tree — "MoE or Dense?"

Appendix C: Year-by-Year MoE Milestones

1. What Is Mixture of Experts?

1.1 The core idea

A neural network in which most of the parameters are not used for any single input. Each input token activates only a small subset of expert sub-networks via a learned router. Total parameter count grows; per-token compute does not.

1.2 Conditional computation

Standard dense networks: every parameter touches every input. MoE: each input touches a tiny fraction of parameters. Frees the model to grow capacity (knowledge, specialization) without paying linearly for compute.

1.3 Active vs total parameters

Two numbers that must be cited side-by-side:

• Total parameters: sum of all weights including all experts.
• Active parameters: weights actually used per token.
• Ratio is typically \(5\text{–}20\times\).

Examples (2025–2026 open frontier):

1.4 Why use MoE?

• Compute-efficient scaling: more knowledge / capacity at fixed FLOPs per token.
• Better quality at fixed inference cost: matches a much larger dense model.
• Training efficiency: faster wall-clock training for a given quality.
• Modularity: experts can specialize; potential for swap-in / swap-out.

1.5 Why not MoE?

• Memory-heavy: all experts must be loaded (or paged).
• Communication overhead: expert parallelism requires all-to-all.
• Routing instability: load imbalance, training collapse.
• Serving complexity: harder to batch; harder to deploy on small machines.
• Quantization is harder: per-expert calibration, outliers.

Key

The 2026 consensus: MoE is the dominant scaling pattern at the open frontier (DeepSeek-V3, Llama 4, Qwen 3, Mixtral 8×22B). For models above \(\sim 30\mathrm{B}\) total, MoE almost always wins on compute-quality Pareto.

2. The MoE Block

2.1 Where MoE goes in a Transformer

Standard placement: replace the FFN / MLP in some or all Transformer blocks with a sparse MoE block.

Attention remains dense.

2.2 The sparse FFN block

Input \(x \in \mathbb{R}^d\), N experts \(\{E_1, \dots, E_N\}\) each a dense MLP \(E_i : \mathbb{R}^d \to \mathbb{R}^d\). Router (gating) \(g : \mathbb{R}^d \to \mathbb{R}^N\) produces routing logits.

\[\mathrm{logits} = W_g x \in \mathbb{R}^N, \quad p = \mathrm{softmax}(\mathrm{logits}).\]

Top-k experts selected:

\[\mathcal{T} = \mathrm{TopK}(p), \quad y = \sum_{i \in \mathcal{T}} \frac{p_i}{\sum_{j \in \mathcal{T}} p_j}\, E_i(x).\]

2.3 Where the router lives

• Per-token routing: each token of the sequence gets its own routing decision (most common).
• Per-sequence routing: rare; loses fine-grained specialization.
• Per-segment / per-document: occasionally for structured inputs.

2.4 How often to insert MoE

• Every layer (Switch, Mixtral): simplest, most capacity.
• Every other layer (some recipes): cheaper, often comparable.
• Last K layers: experimental; less standard.

2.5 Expert size

• Coarse experts: large MLPs (4d hidden) like a dense FFN. Few experts (N = 8). Mixtral.
• Fine-grained experts: small MLPs (e.g., d/4 hidden), many experts (N = 64–256). DeepSeek.
• Trade-off: fine-grained gives more specialization at higher routing overhead.

2.6 Combining outputs

After top-k selection, normalize the gates of selected experts and weighted-sum:

\[y = \sum_{i \in \mathcal{T}} \tilde{p}_i \cdot E_i(x), \quad \tilde{p}_i = p_i / \sum_{j \in \mathcal{T}} p_j.\]

Some recipes skip normalization (use raw \(p_i\)); the difference is empirically small.

3. Routing Strategies

3.1 Top-k routing (standard)

Select the top-k experts by router score. k = 2 standard (GShard, Mixtral); k = 1 also common (Switch). Higher k: more compute per token, possibly better quality.

3.2 Switch routing (top-1)

Switch Transformer (Fedus et al. 2021): k = 1. Each token routed to exactly one expert. Simplest; fastest; surprisingly strong.

3.3 Top-2 routing (Mixtral, GShard)

Standard for many open MoEs. Typically gives a quality bump over top-1 at \(\sim 2\times\) compute.

3.4 Top-k with token dropping

If an expert exceeds capacity, drop the token (it bypasses MoE via residual). Required when capacity is bounded.

Can hurt quality at small capacity; padding adds wasted compute.

3.5 Soft MoE (Puigcerver et al. 2023)

Each expert receives a learned weighted average of all tokens (no hard routing). S slots per expert; S·N slot tokens computed by N experts. Avoids load imbalance and dropped tokens. Used in vision settings.

3.6 Expert Choice routing (Zhou et al.)

Each expert picks the top-k tokens it wants (not vice versa). Automatic load balancing (every expert gets exactly its capacity). Good for training, awkward for autoregressive inference (hard to batch).

3.7 Hash-based routing

Deterministic routing based on token id (hash of input embedding). Eliminates router learning; surprisingly competitive baseline.

3.8 Random routing baseline

Uniformly random expert assignment. Surprisingly hard to beat; ablation reveals routing learns relatively little vs random for some tasks.

3.9 Learned router architecture

• Linear router (single matmul, no bias). Standard.
• Multi-head router (multiple gating decisions averaged). Rare.
• Dot-product with expert embeddings: \(g_i(x) = x^\top e_i\), \(e_i\) a learnable vector per expert. Used in DeepSeekMoE.
• Hash + learn hybrid: hash for stability, learn for refinement.

3.10 Routing temperature

Pre-softmax temperature \(\tau\) sharpens / softens routing distribution:

\[p = \mathrm{softmax}(\mathrm{logits}/\tau).\]

Lower \(\tau\): more decisive (closer to argmax). Higher \(\tau\): more exploration. Often annealed: warmer early in training, sharpen later.

3.11 Router noise

Add Gaussian noise to logits during training:

\[\widetilde{\mathrm{logits}} = \mathrm{logits} + \mathcal{N}(0, \sigma^2).\]

Encourages router exploration; reduces routing collapse early in training.

4. Load Balancing

4.1 The collapse problem

Without explicit balancing, routers tend to concentrate on a few popular experts. Underused experts get no gradient \(\to\) they atrophy \(\to\) even less used. Catastrophic mode collapse to a dense-equivalent network.

4.2 Auxiliary load-balancing loss (Switch)

Let \(f_i\) = fraction of tokens routed to expert \(i\), \(p_i\) = mean router probability for expert \(i\) across the batch. Auxiliary loss:

Key

Switch's load-balancing loss:

\[\mathcal{L}_{\mathrm{LB}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot p_i, \quad \alpha \sim 10^{-2}.\]

Minimizing pushes \(f_i\), \(p_i\) toward \(1/N\) uniform.

The product \(f_i p_i\) is differentiable through \(p_i\) (router output) but not through \(f_i\) (hard top-k). Effective signal is on \(p_i\).

4.3 Z-loss (router z-loss)

Penalize the magnitude of the log-sum-exp of router logits:

\[\mathcal{L}_{\mathrm{Z}} = \alpha_z \cdot \mathbb{E}\!\left[\left(\log \sum_i e^{\mathrm{logits}_i}\right)^2\right].\]

Keeps logits at moderate scale; prevents one expert's logit blowing up. Critical for FP16/BF16 stability.

4.4 Capacity factor

Each expert has capacity \(C = \mathrm{CF} \cdot T/N\) where \(T\) = batch tokens, \(\mathrm{CF} \sim 1.0\text{–}2.0\). When CF is small: token dropping. When CF is large: padding, wasted compute. Trade-off.

4.5 Padding behavior

For batched expert computation, all experts process equal-size buckets. Underused experts pad with zeros (wasted compute). Overused experts drop overflow tokens (quality loss).

4.6 DeepSeek's auxiliary-loss-free balancing

DeepSeek-V3 introduces a per-expert bias term \(b_i\) that's adjusted dynamically (no gradient):

\[\mathrm{score}_i = g_i(x) + b_i.\]

After each step, increase \(b_i\) for under-utilized experts, decrease for over-utilized. Achieves load balance without an auxiliary loss term that biases the gradient. Avoids the quality-vs-balance trade-off.

4.7 Sequence balance loss

Switch / DeepSeek also add a sequence-level balance loss to prevent routing all tokens of a sequence to the same expert:

\[\mathcal{L}_{\mathrm{seq}} = \alpha_s \cdot N \cdot \sum_i f_i^{\mathrm{seq}} \cdot p_i^{\mathrm{seq}},\]

computed per sequence rather than per batch.

4.8 Stability tricks summary

• Auxiliary load-balance loss (or DeepSeek's bias trick).
• Router z-loss for numerical stability.
• Router noise (early training only).
• Capacity factor 1.25–2.0 for safety margin.
• Initialize router weights small (Glorot / Kaiming small variance).
• Stable softmax in router (subtract max before exp).

5. Expert Parallelism

5.1 The problem

MoE total parameters easily exceed single-device memory. Solution: distribute experts across devices. But each token may need an expert on a different device.

5.2 Expert Parallelism (EP)

Each of E devices holds N/E experts. The all-to-all primitive routes tokens to where their experts live, then back:

1. Compute router on every device.
2. All-to-all: send each token to the device hosting its expert.
3. Each device computes its experts on received tokens.
4. All-to-all: send results back to original devices.

5.3 Communication cost

Two all-to-all operations per MoE layer: dispatch and combine. Volume: \(O(B \cdot d/E)\) per device, with \(B\) = global batch tokens.

• Network bottleneck: NVLink within node (\(\sim 900\) GB/s), InfiniBand across nodes (\(\sim 200\) Gb/s).
• Latency: typically the dominant overhead in MoE inference.

5.4 Composing EP with TP/PP/DP

Total devices \(N_{\mathrm{dev}} = \mathrm{DP} \times \mathrm{TP} \times \mathrm{PP} \times \mathrm{EP}\). Common pattern:

• TP within a node (\(\le 8\) devices): for the dense parts (attention).
• EP across nodes: experts distributed.
• DP across replicas: for batch parallelism.
• PP for very large models: pipeline stages.

5.5 Communication-computation overlap

Modern stacks (Megatron-MoE, DeepSpeed-MoE) overlap:

• Token routing computation with previous all-to-all transfer.
• Expert FFN computation with next all-to-all dispatch.

Hides much of the all-to-all latency.

5.6 Expert pinning / sharding choices

• Single-device per expert: simplest, scales linearly to \(E \cdot \text{expert}\) size memory.
• Multi-device per expert (TP within expert): each expert sharded; useful for very large fine-grained experts.
• Dynamic re-pinning: rare; usually static.

5.7 Token dispatch optimizations

• Local-first routing: prefer experts on same node when possible.
• Hierarchical all-to-all: NVLink within node + IB across.
• Gradient accumulation across micro-batches reduces all-to-all frequency.

6. Major MoE Architectures (Open Frontier)

6.1 Switch Transformer (Google 2021)

First production-scale sparse Transformer at trillion params. Top-1 routing, simple load balance, encoder-decoder T5 backbone. Demonstrated MoE matches dense at 1/7 FLOPs.

6.2 GShard (Google 2020)

Top-2 routing across 2048 experts on TPU. Established expert parallelism + auxiliary loss patterns.

6.3 GLaM (Google 2022)

1.2T total / 96.6B active. Decoder-only MoE matched GPT-3 quality at \(\sim 1/3\) training compute.

6.4 ST-MoE

Stable training recipe with z-loss; emphasized stability tricks for MoE at scale.

6.5 Mixtral 8×7B (Mistral 2023)

• 8 experts per layer, top-2 routing.
• 47B total / 13B active.
• GQA, sliding window, RoPE.
• Open-weights; the breakthrough that made MoE accessible.

6.6 Mixtral 8×22B (2024)

141B total / 39B active. Same recipe, scaled. Strong open frontier MoE through 2024.

6.7 DeepSeekMoE / V2 / V3

DeepSeekMoE (2024): introduced two key design choices that became standard:

• Fine-grained experts: many small experts (N = 64+) instead of few large.
• Shared experts: 1–2 always-on experts shared by all tokens; capture common knowledge. Routed experts handle specialization.

DeepSeek-V2 (2024): 236B total / 21B active. + MLA (Multi-Latent Attention).

DeepSeek-V3 (2024): 671B total / 37B active.

• Auxiliary-loss-free balancing (bias trick).
• Multi-Token Prediction (MTP) for training efficiency.
• FP8 training.
• Trained for \(\sim 5.5\mathrm{M}\) H800-hours; reportedly \(\sim\$ 5.5\mathrm{M}\).

DeepSeek-V3.2 / R1 (2025): same architecture; R1 with RL post-training for reasoning.

6.8 Qwen MoE family

Qwen-1.5-MoE-A2.7B, Qwen2-MoE, Qwen2.5-MoE, Qwen3-MoE: Alibaba's open MoE line. Qwen2-57B-A14B and larger; Qwen 3 is 2025 frontier.

6.9 DBRX (Databricks 2024)

132B total / 36B active. Fine-grained 16 experts, top-4 routing (smaller experts than Mixtral). Open-weights enterprise model.

6.10 Grok-1 (xAI 2024)

314B total / 86B active. 8 experts, top-2. Open-weights release.

6.11 Snowflake Arctic (2024)

480B total / 17B active (\(28\times\) ratio — highest among open). Hybrid: dense MLP + 128 small experts in parallel; top-2.

6.12 Skywork-MoE, JetMoE, OLMoE, Phi-MoE, MiniMax-01

Various open MoE releases. OLMoE: fully open including data + training code. MiniMax-01: 456B / 46B active with linear attention + softmax attention hybrid + MoE.

6.13 Llama 4 (2025)

Meta's first open MoE (variant). Scout + Maverick variants; mixture of expert + dense layers.

7. MoE in Vision and Multimodal

7.1 V-MoE (Riquelme et al. 2021)

First large-scale sparse vision Transformer. Per-patch routing in ViT. Showed MoE benefits transfer to vision.

7.2 LIMoE (Mustafa et al. 2022)

Multimodal MoE: shared experts across image and text, with modality-aware load balancing. Per-modality balance loss.

7.3 Soft MoE (Puigcerver et al.)

Soft routing for vision: each expert receives a learnable weighted combination of all tokens. No load imbalance or dropping. Strong on ImageNet at fixed FLOPs.

7.4 MoE-LLaVA

First open MoE VLM. Sparse upcycling from a dense LLaVA: replace some MLPs with N experts initialized from the dense MLP.

7.5 CuMo (CodepathPress 2024)

Vision MoE for multimodal: sparse experts in both vision encoder and MLP projector. Strong VLM performance at lower active compute.

7.6 Aria (Rhymes AI 2024)

24.9B total / 3.9B active. Native multimodal MoE; long-context (64K tokens).

7.7 DeepSeek-VL2 (2024)

27B total / 4.5B active VLM. MoE applied to multimodal; matches much larger dense VLMs.

7.8 Mistral Pixtral, Pixtral-Large

Mistral's multimodal models, some with MoE.

7.9 LLaVA-MoLE, MoVA, Uni-MoE

Various MoE-VLM open lines exploring different routing / expert designs in multimodal context.

7.10 Modality-aware routing

• Per-modality balance loss (LIMoE).
• Modality-specific experts (vision-only, text-only, shared).
• Joint routing where modality embedding is part of the router input.

8. Training MoE Models

8.1 Standard recipe

• AdamW, \(\beta_2 = 0.95\) (vs 0.999 for dense); MoE training noisier.
• LR slightly lower than dense equivalent.
• Warmup over \(\sim 1\%\) of steps; cosine or WSD anneal.
• Auxiliary load-balance loss \(\alpha \sim 10^{-2}\).
• Router z-loss \(\alpha_z \sim 10^{-3}\).
• Capacity factor \(\mathrm{CF} \sim 1.25\) (training); \(\mathrm{CF} \sim 2\) (inference for safety).

8.2 Initialization

• Experts: same init as dense MLP (Kaiming or Xavier).
• Router: small variance \((\sim 0.01)\) to start near-uniform.
• For sparse upcycling (dense \(\to\) MoE): all experts initialized as copies of the dense MLP (with small noise).

8.3 Numerical stability

• Compute router in FP32 even when forward is BF16/FP8.
• Router z-loss is essential at scale.
• Stable softmax in router.
• Gradient clipping \(\tau = 1.0\) standard.

8.4 Auxiliary-loss-free training (DeepSeek-V3 trick)

After each training step:

\[b_i \leftarrow b_i - \gamma \cdot \mathrm{sgn}(f_i - 1/N),\]

where \(f_i\) is the recent fraction routed to expert \(i\), \(\gamma\) small. The bias \(b_i\) added to router scores at selection time but doesn't affect gating weights for combining outputs. Achieves balance with no gradient bias.

8.5 Multi-Token Prediction (MTP, DeepSeek-V3)

Auxiliary head predicts the next 2–4 tokens jointly. Improves data efficiency and can be used for speculative decoding at inference.

8.6 FP8 training for MoE

DeepSeek-V3: FP8 (E4M3) for matmuls in routed experts; BF16 for shared paths and accumulator. Per-block scaling factors. Halves memory; doubles throughput vs BF16.

8.7 Sparse upcycling (dense → MoE)

Convert a pretrained dense LLM into an MoE:

1. Take dense MLPs of layers to convert.
2. Replicate N times (with small noise added).
3. Add a fresh router; train.
4. Retrain on \(\sim 5\text{–}10\%\) of original data; recovers MoE quality.

Cheap path to MoE without pretraining from scratch. Used in MoE-LLaVA, CuMo, Phi-MoE.

8.8 Branch-Train-MiX (BTM, Sukhbaatar et al.)

Train multiple specialist dense experts independently on different domains; assemble into MoE. Avoids the routing-collapse problem entirely.

9. Inference for MoE

9.1 The serving challenge

• All experts must be loadable somewhere (memory requirement).
• Per-token routing breaks naive batching (different tokens want different experts).
• All-to-all communication cost amplified by autoregressive generation.

9.2 Continuous batching with MoE

Continuous batching (vLLM, SGLang, TensorRT-LLM) handles per-token MoE by:

• Routing tokens within the batch independently.
• Grouping tokens by expert assignment for the FFN matmul.
• Result-scattering back to per-request output.

9.3 Expert offloading

For machines with insufficient memory:

• CPU offload: keep cold experts in CPU memory; transfer on demand.
• NVMe / SSD offload: keep cold experts on disk.
• Predictive prefetch: predict next expert from history.
• Used in llama.cpp, MLX, ktransformers for consumer hardware.

9.4 ktransformers / llama.cpp MoE patterns

Make giant MoE models runnable on consumer hardware:

• Load only active experts in GPU; rest in CPU.
• Q4/Q5 quantization on experts.
• Asynchronous expert prefetch.

Enables 671B DeepSeek-V3 on a single workstation with 96GB VRAM.

9.5 Expert quantization

• Per-expert quantization (each expert calibrated separately).
• AWQ, GPTQ, SmoothQuant adapted to MoE.
• Lower bits for cold experts (used less often).
• FP8 (E4M3) inference on H100/B200.

9.6 Speculative decoding for MoE

• Use a small dense draft model + target MoE.
• Or use Multi-Token Prediction head as the drafter (DeepSeek-V3).
• Verification runs the full MoE; speculative tokens accepted/rejected by acceptance probability.

9.7 Caching expert activations

• Hot expert output for repeated prompts.
• Per-token expert assignment cache (not output) for replayed prompts.

9.8 KV cache for MoE

Same as dense (attention is dense). MLA (Multi-Latent Attention) in DeepSeek-V2/V3 dramatically reduces KV memory, freeing budget for experts.

10. Frameworks and Software

10.1 Training frameworks

10.2 Inference frameworks

10.3 MegaBlocks block-sparse kernels

Treats MoE as a block-sparse matmul: each token's expert assignment is a 1-hot in block-sparse B. Custom CUDA kernels run the MoE FFN as a block-sparse GEMM. \(\sim 2\text{–}4\times\) faster than naive token-shuffle implementations.

10.4 Tutel improvements

• Adaptive parallelism (TP \(\leftrightarrow\) EP based on workload).
• Optimized all-to-all dispatch.
• Communication-computation overlap.

11. Specialized MoE Variants

11.1 Mixture-of-Depths (MoD, Raposo et al. 2024)

Sparse compute along the depth axis, not width. Each token can skip layers based on a router decision:

\[y_\ell = \begin{cases} f_\ell(x_\ell) & \text{if router selects this token} \\ x_\ell & \text{else} \end{cases}\]

Different tokens use different effective depths. Saves compute on easy tokens.

11.2 Mixture-of-Recursions (MoR)

Layers can be applied multiple times to hard tokens (recursion); skipped for easy tokens. Combines MoD with iterative refinement.

11.3 MoE in attention

• MoA (Mixture of Attention heads): route to subsets of heads.
• Switch Heads: top-1 head selection.
• Less common than MoE-FFN; quality benefits unclear.

11.4 Conditional computation more broadly

Skip-attention, early-exit (different exit layer per token), pondernet (variable computation per token). MoE is the dominant practical instantiation.

11.5 Hash MoE / Random MoE

Routing is fixed (hash) or random; no learning. Surprisingly competitive. Used as research baseline; rarely production.

11.6 Pyramid Residual MoE (PR-MoE)

Router complexity scales with layer depth; deeper layers have more experts. Cheaper / faster training without quality loss.

12. Expert Specialization and Analysis

12.1 Do experts specialize?

Empirically: for fine-grained MoE (DeepSeekMoE-style), yes — experts develop clear specialization (math, code, document type, language). For coarse Mixtral-style, less clear; experts often look interchangeable.

12.2 Common specializations observed

• Per-language (English / Chinese / code).
• Per-domain (math / wiki / dialogue).
• Per-modality (vision-leaning / text-leaning).
• Per-token-type (whitespace / punctuation / content).

12.3 Shared experts (DeepSeek-style)

1–2 always-on experts shared by all tokens. Capture redundant "common knowledge." Routed experts can specialize because they don't need to relearn the basics. Standard since DeepSeekMoE.

12.4 Expert merging / pruning

• Expert merging: average weights of similar experts; reduce N.
• Expert pruning: drop low-utilization experts; recover via fine-tune.
• MoE \(\to\) dense distillation: train a dense student to match MoE; lose efficiency, gain serving simplicity.

12.5 Routing analysis tools

• Per-expert token-frequency histograms.
• Per-expert input distribution (PCA / t-SNE).
• Per-layer routing entropy (collapse detector).
• Token-expert co-occurrence patterns.

13. MoE Scaling Laws

13.1 Compute-quality scaling

At fixed training FLOPs:

• MoE outperforms dense at the same FLOPs (more parameters used).
• Effective parameter ratio \(\sim 2\text{–}4\times\) at small scale; widens to \(5\text{–}10\times\) at large scale.
• Active params is the main cost driver; total params is the main quality driver.

13.2 Optimal expert count

• N = 8 (Mixtral): simple, easy to balance.
• N = 16–64 (DBRX, DeepSeek): more specialization.
• N = 128+ (Snowflake Arctic, fine-grained DeepSeekMoE): aggressive specialization but harder to balance.

13.3 Optimal k

• k = 1 (Switch): simplest, fastest.
• k = 2 (most): standard, quality bump.
• k = 4+: rare; diminishing returns.
• For fine-grained N, larger k becomes more useful (each expert is smaller).

13.4 Compute-optimal MoE (Chinchilla-style)

For fixed compute C:

• Dense: \(D \approx 20P\) tokens per parameter.
• MoE: more tokens per active param (\(D \approx 30\text{–}50\, P_{\mathrm{active}}\)) because of total-param boost.
• Practical recipes (DeepSeek-V3): \(\sim 14T\) training tokens for 37B active.

13.5 Inference scaling

Per-token inference cost scales with active params, not total. So MoE inference is much cheaper than dense at equivalent quality. Memory cost scales with total params (or use offloading).

14. Production Patterns and Trade-offs

14.1 When MoE wins

• Large-scale training where quality matters more than memory.
• Inference workloads where you can afford the memory.
• Cloud serving: amortize memory across many requests.
• Continuous-batching environments where per-token routing is OK.

14.2 When dense wins

• Memory-constrained edge / on-device.
• Latency-critical small-batch serving (overhead of all-to-all hurts).
• Small models (under \(\sim 7\mathrm{B}\)): MoE's overhead exceeds the gains.
• Quantization-heavy deployments where MoE harder to compress.

14.3 Memory math for MoE serving

For DeepSeek-V3 (671B / 37B active) in BF16:

\[\mathrm{Memory} \approx 2 \cdot 671\mathrm{B} = 1342\ \mathrm{GB}\]

just for weights. With FP8 / Q4 quantization: \(\sim 670\) / \(\sim 335\) GB. Fits on 8×H200 (1128 GB) easily; or 1× workstation with quant + offload.

14.4 Cost models

• Per-token compute: dictated by active params.
• Per-token memory bandwidth: dictated by active params + KV cache.
• Memory footprint: dictated by total params.
• Network: dictated by EP all-to-all volume.

14.5 Comparison: MoE vs Dense (same active params)

At fixed active params, MoE matches a much larger dense model in quality. Mixtral 8×7B (13B active) \(\approx\) Llama-2-70B quality at \(\sim 1/5\) inference cost.

14.6 Multi-tenant serving

• Shared expert weights across requests.
• Per-request routing decisions independent.
• Continuous batching with per-token expert dispatch.
• Expert hot-set caching for popular request patterns.

15. Common Failure Modes and Mitigations

15.1 Routing collapse

All tokens routed to a few experts. Fix: load-balance loss, router noise early in training, capacity factor, DeepSeek bias trick.

15.2 Token dropping

Tokens overflow expert capacity and bypass MoE. Fix: increase capacity factor, balance routing better, or accept (residual carries information).

15.3 Numerical instability

Router logits explode or NaN. Fix: router z-loss, stable softmax, FP32 router, gradient clipping.

15.4 Communication bottleneck

EP all-to-all dominates step time. Fix: hierarchical all-to-all (NVLink + IB), local-first routing, communication-computation overlap.

15.5 Expert atrophy

Some experts get few gradients, never learn. Fix: balance loss, periodic re-init of cold experts, capacity factor.

15.6 Quality regression vs dense

MoE training underperforms expected. Fix: check load balance (entropy), tune \(\alpha\) on aux loss, try sparse upcycling instead of from-scratch.

15.7 Inference batching loss of throughput

Batching breaks because tokens want different experts. Fix: continuous batching at token level, MegaBlocks block-sparse kernels, sufficient parallelism.

16. Recent Frontier (2025–2026)

16.1 DeepSeek-V3 / R1 architecture choices

• Auxiliary-loss-free balancing (bias trick).
• MLA (Multi-Latent Attention) for KV cache.
• 256 routed experts + 1 shared expert per layer.
• Top-8 routing (relatively high k).
• Multi-Token Prediction auxiliary head.
• FP8 training.
• DualPipe scheduling for pipeline efficiency.

16.2 Llama 4 (Meta 2025)

First Llama with MoE; Scout (109B / 17B active) and Maverick (400B / 17B active) variants.

16.3 Qwen 3 MoE

Frontier-open MoE (235B / 22B active and similar). Extended R1-style reasoning post-training.

16.4 MiniMax-01 with hybrid attention

Linear + softmax hybrid attention + MoE; long-context efficient.

16.5 Trends

• Auxiliary-loss-free training becoming standard.
• Fine-grained experts + shared experts (DeepSeek pattern) dominant.
• Higher k for fine-grained (top-6 or top-8).
• MoE in VLMs and multimodal mainstream.
• FP8 / FP4 inference for MoE.

16.6 Open research questions

• Routing learning is fragile; alternatives (hash, expert-choice) understudied.
• Specialization vs interchangeability in coarse vs fine.
• Best practices for sparse upcycling.
• MoE + reasoning / R1: do experts specialize for reasoning sub-skills?
• Compression: how low can per-expert quantization go?

17. Production Stack 2026

★ 2026 SOTA update — Frontier Open MoE LLMs 2025

• DeepSeek-V3.2-Exp: 671B/37B-active + DeepSeek Sparse Attention (DSA) for cheap long context (Sept 2025)
• Qwen3-235B-A22B: unified thinking/non-thinking flagship MoE, 235B total / 22B active
• Kimi K2 (Moonshot): 1.04T total / 32B active agentic MoE trained with the MuonClip optimizer
• GLM-4.5: 355B/32B-active agentic-reasoning-coding MoE; GLM-4.6 follow-up adds 200K context
• gpt-oss-120b / 20b: OpenAI's first open-weight MoE since GPT-2, Apache-2.0, ~5.1B/3.6B active

★ 2026 SOTA update — Trillion-Scale And New Entrants

• Ling-1T (Ant Group): 1T total, ~50B active, 1/32 sparse activation, FP8 training
• Hunyuan-Large (Tencent): 389B/52B-active, then-largest open MoE
• ERNIE 4.5 (Baidu): up to 424B/47B-active heterogeneous multimodal MoE family
• dots.llm1 (rednote): 142B/14B-active fine-grained MoE (top-6 of 128 + 2 shared)
• LongCat-Flash (Meituan): 560B MoE with zero-computation experts, 18.6-31.3B dynamic active

★ 2026 SOTA update — MoE Routing Advances 2025

• Auxiliary-loss-free load balancing (Loss-Free Balancing): per-expert dynamic bias, no interference gradient
• ReMoE: fully differentiable MoE with ReLU routing, drop-in for TopK+Softmax (ICLR 2025)
• Global-TopK / expert-choice successors unifying token-choice and expert-choice balancing
• Zero-computation experts (LongCat): routable no-op experts give per-token dynamic compute

★ 2026 SOTA update — Ultra-Sparse Fine-Grained Designs

• Qwen3-Next-80B-A3B: 80B total / 3B active (~1:27 sparsity), hybrid Gated DeltaNet + MTP
• MiniMax-M2: 230B/10B-active (~1:23) tuned for cheap agentic + coding inference
• Ling-1T: 1/32 highly-sparse activation guided by Ling Scaling Laws
• Fine-grained + shared-expert lineage (DeepSeekMoE) now standard: dots.llm1 top-6/128 + 2 shared

★ 2026 SOTA update — MoE For Reasoning (RL Post-Training)

• DeepSeek-R1: pure-RL reasoning (GRPO) on the DeepSeek-V3 MoE base; published in Nature (Sept 2025)
• Kimi K2 Thinking: 1T/32B-active reasoning agent, native INT4 QAT, 200-300 sequential tool calls
• Ring-1T (Ant): first open 1T-parameter reasoning MoE via asynchronous RL (IcePop stability)
• LongCat-Flash-Thinking: reasoning variant of the zero-computation-expert MoE

★ 2026 SOTA update — MoE Efficiency And Kernels

• NVFP4 pretraining (NVIDIA): 12B model on 10T tokens at 4-bit, matching FP8 quality
• DeepEP: expert-parallel all-to-all comm library with FP8 dispatch/combine for MoE
• DeepGEMM / FlashMLA: FP8 grouped-GEMM and MLA kernels for Hopper/Blackwell MoE serving
• Fiddler: CPU-GPU expert offloading runs Mixtral-8x7B on a single 24GB GPU

★ 2026 SOTA update — Upcycling And MoE Conversion

• Upcycling LLMs into Sparse MoE (NVIDIA): Nemotron-4 15B to MoE, softmax-then-topK finding
• Drop-Upcycling: partial re-initialization for training sparse MoE from dense (ICLR 2025)
• Original sparse-upcycling recipe still the foundation: copy MLP N times + fresh router

Appendix A: Twenty-Five Things to Know

1. Total vs active params: always cite both.
2. Top-k routing: k = 1 Switch, k = 2 standard, k = 8 DeepSeek fine-grained.
3. Switch's load-balance loss: \(\alpha N \sum f_i p_i\).
4. Router z-loss for stability.
5. Capacity factor \(\sim 1.25\) training, \(\sim 2\) inference.
6. DeepSeek's auxiliary-loss-free trick: per-expert bias \(b_i\).
7. Fine-grained experts + shared experts (DeepSeekMoE).
8. Sparse upcycling: dense \(\to\) MoE cheaply.
9. Branch-Train-MiX: independent training \(\to\) assemble.
10. Soft MoE for vision: no hard routing, no token drop.
11. Expert Choice routing: experts pick tokens.
12. Hash MoE: fixed routing baseline.
13. EP all-to-all: dispatch + combine per layer.
14. Composing EP × TP × PP × DP.
15. MegaBlocks block-sparse GEMM kernels.
16. Tutel adaptive parallelism.
17. Continuous batching with per-token expert dispatch.
18. ktransformers for consumer-hardware MoE.
19. DeepSeek-V3: 671B / 37B / FP8 / MTP / MLA.
20. Mixtral 8×7B as the open MoE that broke through.
21. Llama 4 brought MoE into the Llama line.
22. MoE wins at fixed active params; dense wins at fixed memory.
23. Routing collapse fix: aux loss + z-loss + capacity.
24. Mixture-of-Depths: sparse over depth, not width.
25. Shared expert pattern: 1–2 always-on for common knowledge.

Appendix B: Decision Tree — "MoE or Dense?"

1. Memory-constrained device (mobile, edge)? → Dense. MoE memory overhead too high.
2. Single-batch latency-critical? → Dense, unless you can batch enough requests.
3. Cloud serving with high throughput? → MoE. Amortize memory; per-token compute lower.
4. Pretraining frontier model from scratch? → MoE. Compute-quality Pareto wins above \(\sim 30\mathrm{B}\).
5. Have a strong dense LLM, want bigger? → Sparse upcycle dense to MoE.
6. Want easy quantization + serving? → Dense. MoE quantization is harder.
7. Need experts specialized for domains? → Branch-Train-MiX or fine-grained MoE.
8. Vision / multimodal model? → Both viable; DeepSeek-VL2-style or CuMo / Aria if going MoE.

Appendix C: Year-by-Year MoE Milestones

• 1991: Original "adaptive mixtures of local experts" (Jacobs et al.).
• 2017: Sparsely-Gated MoE (Shazeer et al.) — modern MoE in DL is born.
• 2020: GShard (Google); top-2 + auxiliary loss + EP at scale.
• 2021: Switch Transformer (top-1, simple); V-MoE (vision); ST-MoE (stability).
• 2022: GLaM (1.2T params); LIMoE (multimodal).
• 2023: Soft MoE (vision); Mixtral 8×7B (open MoE breakthrough); Expert Choice routing.
• 2024: Mixtral 8×22B; DeepSeekMoE / V2 (fine-grained + shared experts + MLA); DBRX; Grok-1 open; Snowflake Arctic; MoE-LLaVA; CuMo; Aria; DeepSeek-V3 (auxiliary-loss-free + FP8 + MTP).
• 2025: DeepSeek-R1 (MoE + RL reasoning); Llama 4 (MoE); Qwen 3 MoE; MiniMax-01 (hybrid attention + MoE); ktransformers consumer-MoE serving; Mixture-of-Depths.
• 2026: MoE standard at frontier; auxiliary-loss-free training default; FP8 / FP4 MoE inference; multimodal MoE mainstream.