Backpropagation

Last chapter left an open problem: gradient descent needs the slope of the loss with respect to every weight (millions of them), and re-running the network once per weight would be impossibly slow. Backpropagation solves that.

A network isn’t a monolith. It’s a graph of tiny operations: multiplication and addition wired together, then squashed by an activation at the end. Each operation is simple enough to know its own local slope by heart. A multiply knows that nudging one input scales through by the other input. An add passes nudges straight through. A tanh damps them by 1 − tanh².

Backpropagation is bookkeeping over those local facts, plus one piece of vocabulary this book will lean on from here forward. Call a value’s blame the answer to the question: if this value had come out slightly different, how much would the loss have changed? A value with large blame helped cause the error; a value with zero blame didn’t matter. Blame is this book’s plain word for what the math calls a gradient and writes as ∂loss/∂value: the change in the loss per tiny change in this one value, everything else held still. The curly ∂ is chapter 3’s slope idea, asked of one knob among millions.

The algorithm itself is short. Run the graph forward to get the loss; the loss’s own blame is 1, by definition. Then walk backward, and at each node apply one rule, the chain rule: blame arriving at a node’s output, times the node’s local slope, is the blame for its input. By the time you reach the weights, each one holds exactly ∂loss/∂weight, its own share of the blame. One forward pass and one backward pass account for every slope.

That is all backpropagation does: it does not shrink the loss itself. It answers who is to blame, and by how much? so that gradient descent can nudge each weight opposite its blame and drive the loss toward zero. Without those blames, descent has nothing to follow. Without descent, the blames sit unused on the graph.

Step through it yourself:

Watch the backward phase closely and you’ll see the personalities: the square doubles blame, the add copies it to both parents, the multiply swaps in the partner (w₁’s blame gets scaled by x₁: how much w₁ mattered depends on what it was multiplied by), and the tanh damps blame by its slope. When its input is extreme, tanh’s slope is nearly zero, and blame dies there. That’s the “vanishing gradient,” and it returns in chapter 6.

The figure below isn’t an illustration of the algorithm, it ran the algorithm. Every number came from the same ~120-line engine that trains networks throughout this site, and its whole backward pass fits on a napkin:

// value.ts — the heart of it. Each op records how to route blame:
mul(other) {
  const out = new Value(this.data * other.data, [this, other]);
  out._backward = () => {
    this.grad += other.data * out.grad;   // partner swaps in
    other.grad += this.data * out.grad;
  };
  return out;
}

backward() {
  // topologically sort the graph, then apply the chain rule in reverse
  // (the real file inlines the ten-line sort right here)
  this.grad = 1;
  for (const node of reverseTopoOrder(this)) {
    node._backward();
  }
}

# value.py — the heart of it. Each op records how to route blame:
def __mul__(self, other):
    out = Value(self.data * other.data, (self, other))
    def _backward():
        self.grad += other.data * out.grad   # partner swaps in
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

def backward(self):
    # topologically sort the graph, then apply the chain rule in reverse
    self.grad = 1
    for node in reverse_topo_order(self):
        node._backward()

// value.go — the heart of it. Each op records how to route blame:
func (v *Value) Mul(other *Value) *Value {
	out := NewValue(v.Data*other.Data, v, other)
	out.backward = func() {
		v.Grad += other.Data * out.Grad // partner swaps in
		other.Grad += v.Data * out.Grad
	}
	return out
}

func (v *Value) Backward() {
	// topologically sort the graph, then apply the chain rule in reverse
	v.Grad = 1
	for _, node := range reverseTopoOrder(v) {
		node.backward()
	}
}

// Value.java — the heart of it. Each op records how to route blame:
Value mul(Value other) {
    Value out = new Value(this.data * other.data, this, other);
    out.backward = () -> {
        this.grad += other.data * out.grad;   // partner swaps in
        other.grad += this.data * out.grad;
    };
    return out;
}

void backward() {
    // topologically sort the graph, then apply the chain rule in reverse
    this.grad = 1;
    for (Value node : reverseTopoOrder(this)) {
        node.backward.run();
    }
}

We now have neurons that bend (chapter 2), a score to shrink (chapter 3), and every slope. Next: put them together and watch a network train.