Positional Encoding

Attention (chapter 10) computes, for every token, a weighted sum over every other token’s value vector, and the weights come from a dot product between query and key. Nowhere in that computation does the position of a token appear. Shuffle the words in a sentence and feed attention the same set of vectors in a different order, and it produces the same set of outputs in the same shuffled order. Attention treats its input as a set, not a sequence. “The dog bit the man” and “the man bit the dog” would look identical to it: the same six tokens, unless something else tells it who came first.

That something else has to be injected as data, because there’s nowhere else to put it. The original Transformer paper’s trick was to give every position its own fixed fingerprint, a vector, and add it directly onto that position’s token embedding before anything else happens. Position 0 gets fingerprint zero, position 1 gets fingerprint one, and so on, up to however far the model reads. Once that fingerprint is baked into the vector, attention can tell tokens apart by position the same way it already tells them apart by content, because, as far as the matrix math is concerned, position is now content.

The question is what makes a good fingerprint. Binary counting would work in principle: a fixed-size code, one bit per digit. But bits are discrete and jagged, a bad fit for a network trained by smooth gradients. The sinusoidal answer keeps the spirit of binary counting (each dimension flips at its own rate) but makes it continuous: every dimension of the fingerprint is a sine or cosine wave, and each pair of dimensions oscillates at its own frequency, geometrically spaced from fast to slow across the vector. Low dimensions complete a full cycle in a few positions: a fine-grained ruler tick. High dimensions take thousands of positions to complete one cycle: a coarse tick, like the hour hand next to the second hand. Stack all the dimensions together and you get a multi-scale ruler: read a few of a position’s dimensions and you know roughly where it sits in a fine neighborhood; read all of them and you know exactly which position it is, out to very long sequences.

Slide position A and position B around and watch two things. First, the curves: two positions near each other trace almost the same wave, dimension by dimension: a small phase shift, nothing more. Second, the number: cosine similarity between two rows starts high when the positions are close and falls as they drift apart. The scheme is useful not because each position has a fingerprint, but because nearby fingerprints are similar. Attention can learn a rule like “look at the token right before me” using ordinary dot products, the same mechanism it uses for everything else. No special-cased “distance” operation is required. That turns proximity in position into proximity in vector space.

This site’s tiny transformer powering the other chapters’ figures doesn’t use these sinusoids at all. It uses learned position embeddings: one trainable vector per position, initialized randomly and adjusted by gradient descent exactly like the token embeddings sit next to them. That’s simpler to implement and is what GPT-2 and plenty of production models still do; the sinusoidal family above is the classic closed-form scheme from the 2017 paper, included here because it’s the clearest way to see the idea of a positional fingerprint before you just hand the job to backprop. A third family, increasingly the modern default, is RoPE (rotary position embedding): instead of adding a position vector to the embedding, it rotates the query and key vectors themselves by a position-dependent angle before the dot product. It keeps the same “nearby positions look alike” property, but builds it into the attention computation rather than bolting it onto the input.