Part I · Chapter 3 of 18

Loss & Gradient Descent

Roll downhill on the error surface — and learn why step size is everything.

In chapter 1 you tuned weights by hand, eyeballing which slider made things better. To automate that, the machine needs a sense of better: a single number called the loss, scoring how wrong the network currently is. Zero means perfect; bigger means worse. One number the machine can try to shrink.

The classic choice is squared error: for each example, take the difference between prediction and truth, square it (so misses in either direction count, and big misses count a lot), and add them up.

Now imagine computing the loss not just for the current weights, but for every possible setting of a weight. Plot it, with loss on the vertical axis and the weight on the horizontal. You get a landscape, in this case a simple valley. Somewhere at the bottom is the weight that makes the loss smallest. Finding it is a hiking problem: you’re standing on a hillside in fog, and you want the valley floor.

You can’t see the bottom, but you can feel the slope under your feet. Walk downhill:

Roll downhill

step 0 · w = 3.200 · loss = 14.560

learning rate 0.040

The curve is the loss for every possible weight. The ball is where the weight is now. Each step follows the slope — scaled by the learning rate.

Two things to try before reading on. First, run it with the default step size and watch the ball settle into the valley. That’s gradient descent, the algorithm that trains neural networks. Second, crank the learning rate up past ~0.45 and watch the ball overshoot the valley. It ping-pongs between the walls, then flies out of the plot entirely. 💥

That explosion is a failure worth understanding. The learning rate scales each step. Too small: you inch along, wasting time. Too large: each hop overshoots farther than the last, and the loss climbs instead of falling. Every practitioner has watched a training run print loss: NaN because of exactly this. Now you’ve seen it happen, in one dimension, in slow motion.

Real networks have millions of weights, so the landscape isn’t a curve. It’s a surface in millions of dimensions, full of gullies and plateaus. But the algorithm doesn’t change. Feel the slope in every direction at once and take a step downhill, then do it again.

One question remains. Feeling the slope means knowing, for each weight, how the loss would change if you nudged it: millions of weights all feeding that one loss. How can you compute all those slopes without re-running the network millions of times? That method is backpropagation, the subject of the next chapter.

Show the math

Squared-error loss over a dataset: $L(w) = \sum_i (f_w(x_i) - y_i)^2$ . The gradient $\nabla L$ is the vector of partial derivatives $\partial L / \partial w_j$ . It points uphill, so learning steps go the other way: $w \leftarrow w - \eta \, \nabla L(w)$ , where $\eta$ is the learning rate. For a quadratic valley, steps converge when $\eta < 2/L''$ and oscillate or diverge beyond it. The figure’s blow-up is that inequality, live.