Part IV · Chapter 15 of 18

Mixture-of-Experts

Chapter 11 priced two-thirds of a block's weights into one MLP every token pays for in full - what if a token only paid for the sliver it actually needs?

Chapter 11 closed on a ratio worth banking: at typical widths, a transformer block’s parameters split roughly two-to-one in the MLP’s favor. About two-thirds of the weights in every block sit in that one per-token network, only a third in attention, and every token that passes through pays the entire cost of that MLP, every layer, every time. The plain word “the” gets multiplied through the exact same 32,768-weight network as a token doing hard semantic work. Every token gets the same one-size-fits-all treatment, whether or not the size actually fits.

What if each token only activated the MLP capacity it actually needs? Mixture-of-experts takes that question literally. Instead of one MLP per block, keep many (call them experts, each the same shape as the MLP it replaces) and add a small router, a tiny linear layer that looks at a token’s vector and scores every expert for “how much should you handle this one.” Send the token to only its top few scorers, weight each one’s output by how confident the router was, and add the results together, so most experts never touch that token at all. Total parameters grow with however many experts you add: that’s just more weights sitting in memory. Total compute per token grows only with how many experts actually fire. Decoupling those two numbers, so a model’s knowledge capacity can scale far faster than its per-token compute bill, is what this chapter is about.

Mixture-of-experts routing, live

loading the model (≈1.2 MB)…

text

Honest note: this router's gate matrix is a random, untrained 64→8 projection — it was never optimized to route anything. Its embeddings are real, though (the trained model's own token vectors), so the routing you see here is the raw mechanics of the mechanism, not a demonstration of what a trained router would learn to do.

The router above is deliberately untrained: a fixed random projection, not the product of a training run, so what you’re watching is the mechanism, not a demonstration of routing wisdom. Click through a few tokens and one thing is worth noticing. On the default sentence, «Queen», «Alice», and «said» (the two proper nouns and the verb tying them together) all land on the identical top-2 pair of experts; «The» shares only one of those two, and an unrelated fragment like «un» shares neither. The gate matrix never learned anything, but the embeddings it’s scoring did. Real structure surviving from chapter 9’s trained embedding space still bends an untrained router’s picks, just not into anything resembling what a trained router would learn to do with them. Every token lands on some pair of experts with a gate percentage attached, and the aggregate load bars underneath make the imbalance a random router produces obvious at a glance: with nothing pushing back, a handful of experts end up popular and the rest sit nearly idle. Toggle the FLOPs readout and the payoff shows up as exact numbers on this toy’s own scale: 8 experts built at chapter 11’s MLP shape cost 262,144 multiplies per token if every one of them fired, but only 2 of them actually do, so the real bill is 65,536, a clean 4× cut, paid against the identical 262,144-multiply pool of parameters that never shrank at all.

That imbalance is the first entry in MoE’s engineering tax. Left alone, a router converges on favorites: a few experts get all the practice and all the gradient signal, and the rest undertrain and become dead weight, which is worse for capacity than not having them. Production systems fight this with an auxiliary loss that penalizes lopsided routing, or, in DeepSeek-V3’s case, a bias term added directly to the routing scores and nudged up or down after each batch, no extra loss term to balance against the main objective, just pressure toward even load. Balance is the first tax. The second is structural: parameters may be sparse per token, but every expert still has to live somewhere in GPU memory, all the time, whether or not it’s firing this step. Large MoE models shard experts across GPUs and route each token’s chosen experts across that mesh with an all-to-all communication step, every MoE layer, every forward pass. That’s real network traffic standing in for what would, in a dense model, just be a local matrix multiply.

The payoff is real enough that every frontier lab eats that tax anyway. Mixtral 8×7B, Mistral’s openly published model, popularized the recipe at accessible scale: 8 experts per MLP layer, top-2 routing, 46.7B total parameters but only about 12.9B active on any given token, under a third of the weight count doing the work, by design. DeepSeek-V3 pushed the same idea to frontier scale and published the exact shape: 256 routed experts plus 1 always-on shared expert per layer, top-8 routing, 671B total parameters, 37B active per token. Kimi K2, from Moonshot AI, lands in the same published-family neighborhood, roughly 1 trillion total parameters, about 32 billion active per token. All three models make the same trade for the same reason: park enormous knowledge capacity in weights that mostly sit still, and only wake the fraction of it that a given token actually needs.

Show the math

One MoE layer’s output is a sparse weighted sum over experts: $y = \sum_{i \in \text{top-}k} g_i \, E_i(x)$ , where $E_i$ is the $i$ -th expert (an ordinary MLP), $g_i$ is that expert’s router weight for this token (softmax over the router’s scores, restricted to the chosen $k$ ), and every expert outside the top- $k$ contributes exactly zero: not a small amount, zero, because it never runs.

The number worth watching is the active/total ratio $k / n$ , where $n$ is the total expert count: it’s the exact fraction of the model’s parameters any single token’s forward pass touches. This toy uses $k=2$ , $n=8$ : a ratio of $1/4$ , matching the figure’s measured 4× compute cut directly, since compute scales with $k$ while the parameter pool scales with $n$ . DeepSeek-V3’s published $37\text{B} / 671\text{B} \approx 5.5\%$ is the same ratio at frontier scale. It’s smaller because frontier routers pick fewer of many more experts, not because the idea changes.

For engineers

Mixture-of-experts is sharding you can’t opt out of. The moment a model has more expert weights than fit comfortably on one accelerator, expert placement across a cluster stops being a deployment detail and becomes part of the architecture: which experts live where, and how much all-to-all bandwidth a training step burns moving tokens across the mesh before it computes a single useful FLOP. Router collapse, the imbalance this chapter’s figure showed you happening for free with zero training pressure against it, is the ops pain that keeps that sharding honest. A lab that ships an MoE model ships a load-balancing story right alongside it, whether that’s an auxiliary loss or DeepSeek’s bias-based alternative.

Keep this chapter’s ratio in mind when reading spec sheets. “1 trillion parameters” is a real number and a genuine headline, but it answers a different question than “how much compute does one forward pass cost.” The honest second number (DeepSeek-V3’s 37B active, Kimi K2’s roughly 32B, Mixtral’s 12.9B) is what predicts latency and serving cost, plus GPU footprint. Total parameters measure how much the model knows; the active count is what you pay when you ask it something. Read both whenever either one gets quoted alone.