The Training Playground

Everything from Part I comes together on this screen. The network you are about to train is a multi-layer perceptron, or MLP: neurons arranged in columns, with every neuron wired to every neuron in the next column. It is the plainest neural network there is.

The columns in the middle are hidden layers, hidden because you never read their values directly. They are working notes the network keeps for itself. The playground below offers two of these shapes, and the diagram covers both. Set hidden layers to 1 and you have its 8 neurons network. Leave it at 2 and you have 8 × 8 deep.

That network is below with real weights and real tanh neurons, running on the autograd engine from last chapter. The colored regions are its current predictions across the plane, and the dots are the labeled data. Press Train and watch the boundary move toward the labels.

Things worth trying:

1. XOR. It starts on the dataset a single neuron cannot solve (chapter 1). Press Train. Within a few hundred epochs, the boundary folds into the four-quadrant checkerboard no single line could draw. That takes about two seconds once you have layers of tanh neurons.
2. Circles. A ring around a disk. The network has to learn a closed curve: early on the boundary is a loose blob, then it tightens around the inner cluster.
3. Spiral, and the two network sizes. Leave 8 neurons selected and train the spiral. One hidden layer struggles to wrap two interlocking arms. Then switch to 8 × 8 deep and compare. Some shapes need more than one hidden layer, and that is half of chapter 6.
4. Push the learning rate. Set it to 1.0. There is no chapter-3 explosion: tanh caps every prediction inside (−1, 1), so this loss surface has no cliff. On these small, full-batch problems, the high rate usually wins (try Spiral at 1.0 versus 0.1; the fast run reaches a loss the slow one won’t). Don’t over-learn that lesson: real training uses noisier gradients and unbounded losses, where chapter 3’s 💥 sits one high step away. The safety here comes from the architecture, not a general rule.
5. Watch the loss curve, not just the picture. The sparkline is what practitioners monitor. Smooth descent means learning. A fast drop that then stalls often means the network is too small for the shape (try Spiral on 8 neurons).

This screen puts Part I together: weighted sums (ch. 1), tanh (ch. 2), loss driven downhill (ch. 3), and backprop supplying every gradient (ch. 4). The training loop is the same one used at large scale. What changes with scale is the size of the network, not the steps.

Part II asks what happens when you stack many layers (and why gradients vanish), and how to shape the network to match the data: grids for images, loops for sequences.