Part I · Chapter 2 of 18

Activation Functions

Why networks need to bend — and what happens when they can't.

Chapter 1 left us stuck: a neuron’s decision is a straight line, and XOR needs something a straight line cannot draw. Your first instinct might be to throw more neurons at the problem: feed the outputs of two neurons into a third. Stacking linear things does not help. A weighted sum of weighted sums is still a weighted sum. Pile up a hundred layers of pure w·x + b and the whole tower collapses, mathematically, into one straight line: the same limit as before, reached with more arithmetic.

Somewhere in that stack, one step has to not be a weighted sum. That step is the activation function, the σ wrapped around every neuron’s sum. Chapter 1’s neuron used the bluntest version there is: a hard step from 0 to 1. Meet the rest of the family:

The activation zoo

f(0.80) = 0.66

input x 0.80 show slope

Pick a function, slide the input. The derivative (toggle) is what chapter 3 will climb along.

Each has a personality:

Step was the 1958 original: decisive and binary. As you may have discovered with the “show slope” toggle, it is also unteachable. Its slope is zero everywhere, so it offers no hint about which direction would improve things.
Sigmoid smoothly squashes anything into (0, 1), and its slope is alive near the middle. The classic choice for probability-flavored outputs.
Tanh is sigmoid’s centered sibling, squashing into (−1, 1). Outputs hover around zero, which keeps layered networks better behaved. It’s what our engine uses by default.
ReLU is simple: negative in, zero out; positive in, unchanged. The workhorse of deep learning, mostly because it’s cheap and its slope doesn’t fade at large inputs the way the squashers’ do.

Every useful activation has a meaningful slope somewhere, and that slope is what learning runs on. Training keeps asking one question of every neuron: if your sum shifts a little, which way does your output move, and by how much? Sigmoid, tanh, and ReLU give real answers, and those answers are what weight adjustment steers by, as the next two chapters show. The step’s answer is always “not at all”: its slope is zero everywhere, so no directional signal survives passing through it, and nothing behind a step neuron can learn.

Show the math

Composition of affine maps is affine: $W_2(W_1x + b_1) + b_2 = (W_2W_1)x + (W_2b_1 + b_2)$ , which collapses to one matrix and one bias, still just a line. Nonlinearity between layers breaks that collapse. The derivatives that matter later are $\sigma'(x) = \sigma(x)(1-\sigma(x))$ for sigmoid, $1 - \tanh^2(x)$ for tanh, and $\mathbb{1}[x > 0]$ for ReLU.

The bend and the slope do two different jobs. The bend buys expressive power: stacked layers can now carve curved boundaries, so XOR is solvable. The slope buys teachability. Learning, which the next two chapters build, will ask of every weight whether a small nudge up or down makes the answer better, and the activation’s slope is a link in the chain that carries that answer back to the weight. A zero slope, like the step’s, snaps the chain. Before any of that we need a way to score how wrong the network is, and then a way to roll downhill on that score. That’s the next chapter.