Part II · Chapter 7 of 18
Convolutional Networks
Stop wiring every pixel to every neuron - slide one tiny neuron everywhere instead.
Feed a photo to the networks we’ve built so far and you hit a scaling problem. A megapixel image has a million inputs, and wiring each to even a modest hidden layer costs billions of weights, most of which would relearn the same lesson. A dense layer doesn’t know that an edge in the top-left corner is the same thing as an edge in the bottom-right. Images have structure: nearby pixels matter to each other, and patterns repeat everywhere. A layer that ignores that structure pays for its ignorance in parameters.
Convolution solves this by building one tiny neuron (a 3×3 grid of weights called a kernel) and sliding it across the whole image, computing its little weighted sum at every position, reusing the same nine weights everywhere it lands. The grid of responses it leaves behind is a feature map: a picture of where the pattern lives. Watch it happen:
Press Step to walk one position at a time, or Play to run all 144.
Left: the input image. Right: the feature map — one cell per kernel position. Blue = positive response, red = negative.
Play the vertical-edge kernel across the window frame. Look at its weights: negative column, zeros, positive column. It computes “right side minus left side,” so it fires exactly where brightness changes left-to-right: the frame’s vertical strokes. Switch to horizontal edges and the response flips to the other strokes. Same image, different kernel: a different question asked at every location. Sharpen and blur are the same machinery wearing photo-editing clothes; your phone’s filters are convolutions.
The savings show up once you count the weights: that kernel asked its question at 144 positions using 9 weights total. A dense layer answering the same 144 questions would have needed 144 separate neurons × 196 inputs ≈ 28,000 weights. Weight sharing is a ~3000× discount, and it comes with a structural bonus: the pattern detector works everywhere by construction, so the network doesn’t have to see a cat in every corner of the frame to recognize corner cats.
A real convolutional network stacks this: dozens of kernels per layer (each learning its own pattern, and yes, learning: kernels are trained by the same backprop as everything else, not hand-designed like our presets), with each layer’s feature maps becoming the next layer’s input. Layer one finds edges; deeper layers combine those edges into corners, textures, eyes, wheels, faces. It’s chapter 6’s composition story, but with geometry built in.
Show the math
A convolution computes , a dot product between the kernel and each patch. Our figure uses “valid” padding ( outputs per side) and stride 1. Parameter count for a conv layer: , independent of image size, which is why convolution scales where dense layers don’t.
One more architecture question remains for Part II. Images repeat in space, but what about data that unfolds in time, where the past has to inform the present? That needs a network with memory.