RNNs & LSTMs

Everything so far looks at its input all at once. But language, audio, sensor logs, and keystrokes are sequences: they arrive one piece at a time, and the meaning of now depends on before. “The trophy didn’t fit in the suitcase because it was too big.” Resolving that “it” requires remembering two nouns from earlier in the stream.

The economical answer is a recurrent neural network, or RNN: one set of weights processes each time step, and alongside the input it receives its own previous output, a hidden state h, the network’s running summary of everything so far. The figure below is the smallest version of that loop. The blue bars are h at each of twenty time steps: how strong the running memory is right then. The bottom row is the input. Click a cell to place or clear a spike, a one-step pulse of input (the red triangle). The recurrence w slider sets how much of last step’s h is kept when forming this step’s h.

Start from the default: one spike early in the sequence. At w = 0.65, each step keeps only 65% of the previous h, so after the spike the bars shrink and that memory is gone within five steps, long before step 20. Slide w toward 1.0 and the bars stay tall longer. Push past 1.0 and the loop feeds itself faster than tanh can squash it: h locks into a self-sustaining value that never fades.

Now connect this to training. Write the loop out as twenty successive steps (unrolling it) and you get a 20-layer network in which every layer shares the same weight. Chapter 6’s vanishing gradient returns in a sharper form: blame flowing back from step 20 to step 2 crosses eighteen copies of the same multiplication. When that factor sits below 1, the signal vanishes exponentially. When it sits above 1, the signal explodes. The figure’s knife-edge at w ≈ 1 is exactly this training problem. Plain RNNs can’t hold memories across long gaps because gradients can’t survive the trip back to deposit them.

The 1997 fix is the LSTM (Long Short-Term Memory). Instead of hoping the loop’s weight lands near 1, it takes control of the loop: alongside h it keeps a protected cell state, a conveyor belt for memory, governed by learned gates. A forget gate decides what fraction of the old memory survives (a learned, per-moment version of our w slider). An input gate decides what new information gets written. An output gate decides what gets revealed. The gates are tiny neurons themselves, so the network learns when to remember and when to forget, instead of decaying at a fixed rate. On the conveyor belt, gradients travel back through addition, not repeated multiplication. The long-gap memory problem is largely solved.

For fifteen years LSTMs were the default for sequences: translation and speech recognition were built on them, and so was the autocomplete on your phone. But notice the shape of what they do. All of history is squeezed through one fixed-size hidden state, step by step. What if, Part III asks, instead of carrying a compressed memory forward, the network could simply look back at any part of the sequence, directly, whenever it needs to?