Part IV · Chapter 17 of 18
Anatomy of a Frontier Model
Sixteen chapters, each one box in the blueprint - assemble them per the published DeepSeek-V3 / Kimi-K2 family recipe and the tower reads like an ordinary system diagram.
Chapter 15 closed on a headline worth re-reading: “1 trillion parameters” and “32 billion active” are both true of the same model, and neither number alone tells you what you need to know. Chapter 16 closed on a related point: “1 million token context” is a systems achievement wearing a modeling-sounding name. Both chapters were describing pieces of one machine. This chapter puts the pieces back together. Nothing new gets introduced from here: no new formula, no new mechanism. This is the assembly chapter, where sixteen chapters’ worth of boxes get stacked in the order a real frontier stack stacks them, and you get to read the result the way an engineer reads a system diagram: component by component.
Representative of the published DeepSeek-V3 / Kimi-K2 family recipe — frontier labs' exact production internals (K3, Fable-class) are not public.
Embedding Byte-level BPE tokenizer, vocabulary ≈160,000 — the same lookup-table idea chapter 9 introduced at three-novel scale, re-tuned for a training corpus spanning the open web. Every token still becomes one vector in one big table; only the size of the table changed on the way to a frontier model.
Select a region for its published-family numbers and the chapter that explains it.
Walk the figure top to bottom and every region has a name you already know. Embedding is chapter 9’s lookup table, unchanged in kind, just re-tuned to a byte-level BPE vocabulary of roughly 160,000 entries so the model can represent the open web instead of three Gutenberg novels. Each of the 61 stacked blocks is chapter 11’s transformer block (attention, residual, MLP, residual, the exact shape) with two of its interior parts swapped for the frontier-scale versions this book’s last two chapters built up to. Attention is chapter 10’s softmax(QKᵀ/√d)V, wearing chapter 16’s fix: multi-head latent attention compresses each token’s key and value into one shared ≈512-dimension latent instead of caching them at full size, because a KV cache that scales with 61 layers and a 128k context window has to be compressed or it will not fit in GPU memory. The block’s MLP is chapter 15’s mixture of experts: 1 shared expert plus 256 routed experts standing in for the single dense MLP chapter 11 described, a router sending every token to its own top 8, so parameter count and per-token compute finally stop being the same number. And at the top of the stack, the LM head is chapter 13’s next-token projection, aimed at that same ≈160k-entry vocabulary, trained in part with chapter 14’s pipeline plus a multi-token-prediction objective layered on top: predict a couple of tokens ahead instead of strictly one, which published results credit with a stronger training signal during pretraining and, as a side benefit, a faster decoding trick at inference. Click through the figure’s four regions and each fact panel names its chapter explicitly; none of this should feel unfamiliar, because none of it is new.
That “published-family” qualifier attached to every number in this figure is the actual boundary of what this chapter can show you. DeepSeek-V3 and Kimi K2 are real models with real published technical reports: their authors disclosed architecture shapes, parameter counts, and training-objective descriptions in enough detail to draw the diagram above with a straight face. What they did not disclose, and what no frontier lab discloses in full, is everything downstream of “here is the shape”: the exact training data mixture and its filtering rules, the specific reward models and RL hyperparameters behind post-training (chapter 18 opens that door as far as it honestly goes), the internal eval suites a lab uses to decide a model is ready to ship, and any of it for models whose architecture was never published at all. Kimi K3, or systems built by labs like Anthropic, are simply not public in the way DeepSeek-V3 and Kimi K2 chose to be. This figure is the published family recipe, assembled from real disclosed numbers. It is not a photograph of any specific production system’s internals, and no honest diagram of one exists outside the labs that run it.
Read that boundary correctly and the claim gets clearer. Every box in the figure above is a chapter you’ve already read in full, with working code behind it: an embedding table, a softmax, a residual stream, an MLP (or 261 of them wearing a router’s decision), and a linear layer predicting what comes next. A frontier model is not a different kind of thing from the single neuron chapter 1 opened with. It is that same weighted sum, stacked wider and sparser, trained on a scale that turns “unfamiliar” into “expensive,” but never into “incomprehensible.”
Reproducing ≈1T total / ≈32B active — the coarse arithmetic
Take the figure’s shape at face value and do the multiplication a spec sheet is hiding. All inputs below are published-family approximations, not any single model’s disclosed internals to the last parameter. The point is that the arithmetic is ordinary multiplication, not that these exact digits are anyone’s filing.
Assume each expert (shared or routed) is an MLP of roughly parameters (a published-family-scale guess for a single expert at this width). Every layer holds shared routed experts, and there are layers:
That single term already lands on the headline number, which is the point chapter 15 made about ordinary dense MLPs, now visible at trillion-parameter scale: the expert pool is the model, weight-count wise. Embeddings, attention weights, and the LM head add only a small sliver on top, a few billion parameters, not enough to move the “≈1T” rounding.
Active parameters follow the same multiplication, restricted to what actually fires per token: 1 shared plus the top-8 routed experts, not all 257:
Add attention’s own weights (always active, since MLA isn’t routed) and the total lands in the low-to-mid tens of billions, the same neighborhood as the ≈32B commonly quoted for this published family. The few billion of daylight between this estimate and the published figure is exactly what “coarse, approximate accounting” should produce: real deployments spread routed capacity across more (and individually smaller) experts than this figure’s round 256, among other simplifications. What survives the imprecision is the ratio: , the same kind of sparsity chapter 15 measured on a toy of 8 experts, now measured on the real thing.