Tokens & Embeddings

Chapter 8 ended by asking whether a network could look back at any part of a sequence directly, instead of carrying a compressed memory forward. That look-back still needs something to look at. A neuron only computes on numbers; it has no idea what a letter is. So the sentence has to become a sequence of numbers first.

A token is one chunk of text the model treats as a single unit, and the vocabulary is the fixed list of chunks it knows. There are two naive ways to choose them, and both break.

Feed the network individual characters. The vocabulary stays tiny (a few dozen symbols) and nothing is ever out-of-vocabulary, meaning absent from the list. But a paragraph explodes into hundreds of tokens, and every one of them is nearly meaningless on its own: the network has to re-discover that “t-h-e” is a word before it can think about what the word means.

Feed it whole words instead. Now each token actually means something, but the vocabulary grows without bound. Every name or typo needs its own slot, and so does every invented word, while words that clearly share structure (“run”, “running”, “runner”) get completely unrelated ids, so nothing learned about one transfers to the others.

Byte-pair encoding (BPE) splits the difference by learning the vocabulary from data instead of guessing it in advance. Start with individual characters as the only tokens. Scan a large body of text and find the pair of adjacent symbols that occurs together most often, say t and h. Merge every occurrence of that pair into a single new symbol, th. Repeat: find the next most frequent adjacent pair (maybe th and e, giving the) and merge it too. Do this a few hundred times and useful structure appears. Strings that show up constantly, like “the” and “ing”, earn their own single token, while anything rare stays broken into smaller pieces. Nothing is ever truly out-of-vocabulary, because in the worst case a word falls all the way back to individual characters.

Type into the text box and watch the colored chips below: each chip is one token. Common words like “the” or “and” usually stay as a single chip. Try something rarer next to a common word (“extraordinarily” beside “the” works well) and watch the rare word break into three or four chips this vocabulary happened to learn were worth keeping.

Tokenizing gets text down to a sequence of integer ids, but an id carries no relationships: id 47 isn’t closer to id 48 than it is to id 9000. The next step is to give every id a learned vector, a list of numbers stored as a row in a big trainable matrix and looked up by id. That vector is the token’s embedding. Nothing about it is hand-designed. It starts random, and the only pressure on it is the network’s training objective, predicting the next token, applied over and over across a whole corpus. If two tokens tend to show up in similar contexts, the gradient nudges their vectors in similar directions each time, and over millions of updates they drift toward each other in the embedding space. In this scheme, “meaning” is geometry that training discovered on its own.

This is the real embedding space this site’s tiny model learned, flattened from 64 dimensions down to 2 so it fits on a page. The flattening keeps some structure and loses some. Use the search box to find a few tokens and see what lands near what. Punctuation marks tend to sit near other punctuation, and pronouns near other pronouns. Dialogue verbs like “said” and “cried”, the words that report how something was spoken in these storybooks, tend to keep close company too. Nobody told the model that punctuation is a category or that “cried” is a cousin of “said.” It fell out from which tokens kept appearing in similar neighborhoods.

This site’s tokenizer has a vocabulary of 383 tokens and 64-dimensional embeddings, trained on three short storybooks. Production language models use the same recipe at a very different scale: vocabularies around 100,000 tokens, embeddings thousands of dimensions wide, and training data spanning a meaningful fraction of the internet. Only the matrices and the data pushing on them get bigger.