Part II · Chapter 6 of 18

Going Deep: MLPs

Stacking layers buys expressive power - and quietly starves the layers farthest from the loss.

Part II is about shaping what Part I built, starting with the multi-layer perceptron you just trained. In the playground, most of the toy datasets yield to a single hidden layer of eight neurons. The spiral does not. Its two interlocking arms need a boundary that twists along their length, so the one-layer network stalls on a coarse blob where two layers of eight wrap the arms. This chapter explains why depth helped, and then looks at the problem depth creates.

Why does depth help at all? One hidden layer is already enough to fit any shape, given enough neurons, but look at how it gets there. Each tanh neuron draws one soft step across the plane, dividing it into a high side and a low side. Place two opposing steps close together and their sum is a bump: a patch that is active in one narrow band and quiet everywhere else. A one-layer network fits an awkward boundary by tiling it with patches like these, and what it learns for one patch does nothing for the next. The spiral needs patches all the way along both arms, so the neuron count climbs with how intricate the shape is.

A second layer changes what the network is allowed to look at. It never sees the raw coordinates, only the numbers the first layer produced. So the first layer can settle on a few reusable features, such as which side of a given line a point falls on, and the second layer can combine those features into answers, using each one in many places. Building later answers out of earlier parts is what composing means here, and it’s the same reason you write functions instead of one enormous main().

Both of those choices have a name. Width is how many neurons sit in one layer, the “neurons per layer” slider on the last chapter’s diagram. Depth is how many hidden layers you stack, its “hidden layers” slider. Width adds real capacity, since a wider layer can hold more features at once. What depth adds is the reuse just described, because only a later layer can build on an earlier one’s output. Most problems want some of each, and depth usually matters more.

Depth has a cost, and it lands on the backward pass. Remember how blame travels (chapter 4): backward through every operation, multiplied by each local slope and weight along the way. Passing back through one layer multiplies the gradient by some number, and that number is the layer’s factor. Eight layers means eight factors multiplied together. Say each one is 0.5, which sounds mild enough. Multiplied out across eight layers that leaves about 0.004, so the first layer hears roughly a 250th of what the last one hears. Smaller factors collapse faster, exponentially so as depth grows. The old textbook advice guaranteed small factors without meaning to. It said to initialize with small random weights, and a weight well below 1 times a tanh slope that never exceeds 1 leaves every factor below 1. Watch what that does to a deep stack:

How much of the learning signal reaches each layer

layer 1 is corrected 1.6k× more weakly than the output layer, and 5 layers are under 1% (shown in red)

Each layer multiplies the passing blame by weights smaller than 1 and a tanh slope of at most 1. 8 hidden layers deep, the product has collapsed — the layers that must learn the most basic features are effectively frozen. Now flip to fan-in scaled and re-roll: same architecture, same randomness, factors sized to 1.

hidden layers 8

One backward pass. Each bar is the average size of the gradient landing on that layer's weights, measured against the output layer's — so the output bar is 1× and a bar at 1/1k means that layer is being corrected a thousand times more weakly. The scale is logarithmic: each step down the axis is a factor of ten.

The figure starts eight hidden layers deep, with the classic small random weights. Each bar is the learning signal reaching one layer, measured against what the last layer gets, and Re-roll weights draws a fresh random start. The pattern survives every draw: layer 1, the layer that must learn the most basic features of the input, hears the loss at about a thousandth of the volume the output layer gets, which for practical purposes freezes it. Drag the slider down to two or three layers and the gap nearly closes, which is why shallow networks trained perfectly well all along. This is the vanishing gradient problem, and it is why “just stack more layers” failed for a decade: the layers expected to learn the most were the ones feedback could no longer reach.

Now flip to Fan-in scaled and re-roll. Same architecture, same randomness, but each weight is sized against how many inputs its neuron sums (Xavier initialization, what every framework quietly does today), so each layer’s factor in the product sits near 1. The bars level out on every draw. The architecture and the math are unchanged. The only difference is the numbers the weights started from.

The other fixes act on the same product. Activations with slope 1 (ReLU, from chapter 2) stop the slopes from shrinking the product further, which is half of why ReLU became the default. Weights sized too large produce the mirror-image exploding gradient (the math aside has the condition). Skip connections add an identity path around a group of layers, which lets blame bypass those multiplications entirely. They arrive in Part III. Every one of these fixes is a way of holding that product of numbers near 1.

Show the math

The gradient reaching layer $k$ of an $n$ -layer net is a product of Jacobians: $\frac{\partial L}{\partial h_k} = \frac{\partial L}{\partial h_n}\prod_{j=k}^{n-1} W_{j+1}^\top \,\mathrm{diag}(\sigma'(z_j))$ . Write $\gamma_j = \lVert W_{j+1}^\top \,\mathrm{diag}(\sigma'(z_j)) \rVert_2$ for the size of one layer’s factor, so that $\lVert \partial L/\partial h_k \rVert \le \lVert \partial L/\partial h_n \rVert \prod_{j=k}^{n-1} \gamma_j$ . If every $\gamma_j \le \gamma < 1$ , the gradient decays at least as fast as $\gamma^{\,n-k}$ , which is the vanishing case. Growth therefore requires $\gamma_j > 1$ somewhere, and because $\gamma_j \le \lVert W_{j+1} \rVert_2 \cdot \max_j |\sigma'(z_j)|$ with $|\sigma'| \le 1$ for tanh, it requires $\lVert W_{j+1} \rVert_2 > 1$ : the weight matrix has to more than make up for the slope. That condition is necessary rather than sufficient, since saturated units drive $\sigma'$ toward zero and can hold $\gamma_j$ below 1 even when the weights are large. With ReLU, $\sigma' \in \{0, 1\}$ , so active paths pass the weight term through unchanged.

Convolutional networks, next chapter, take composition a step further: instead of just stacking layers, they shape each layer to match the structure of the data itself.