Transformers and Attention Is All You Need

Type anything into a chatbot, and underneath the conversation is one repeated operation: given the text so far, guess what comes next.

That guess is not a single word. It is a probability spread across the model's entire vocabulary, every possible next token ranked by how likely it is.

Everything else on this page, tokenization, embeddings, attention, stacked layers, is really just how that probability distribution gets computed.

Before any of that math happens, text is split into tokens, roughly word-sized pieces.

Each token is looked up in a table, learned during training, that maps it to a fixed-size vector of numbers: its embedding.

That lookup table has no idea about context. The same token always starts from exactly the same vector, no matter what surrounds it.

"Roughly" word-sized is doing real work in that first sentence: a real tokenizer only keeps common words whole. A rarer or unfamiliar word gets split into multiple subword pieces instead, which is a big part of why LLMs sometimes stumble on letter-counting or unusual spellings, the model never sees that word as one atomic unit to begin with.

"river bank" and "savings bank" both contain the token "bank". Its starting embedding is identical in both phrases.

But the two phrases clearly need different meanings for that word. A static, one-size-fits-all vector cannot capture that by itself.

What the model needs is a way to let each token's vector shift based on the tokens before it, pulled toward whichever meaning the earlier context supports. (Both examples here put the disambiguating word first, on purpose: a decoder can only look backward, never forward, as the next section covers.)

Every token's embedding produces three new vectors: a Query (what this token is looking for), a Key (what this token contains), and a Value (what it offers if attended to).

Each is produced by multiplying the embedding by its own learned weight matrix, the same three matrices reused for every token in the sequence.

The Query and Key vectors are what get compared to decide attention weights. The Value vector is what actually gets passed forward once those weights are set.

For every pair of tokens, take the dot product of one token's Query with the other's Key. A high dot product means that Key looks like what the Query is searching for.

Those raw scores are divided by the square root of the key dimension. Without this, large dot products push softmax into regions where its gradient is nearly flat, which makes the model much harder to train.

Since this model is predicting the next token, a token is also blocked from attending to anything that comes after it. Those positions are masked to negative infinity before softmax runs, so they get exactly zero weight.

Softmax turns each token's row of remaining scores into a proper probability distribution: every value positive, the whole row summing to 1.

Do this for every token at once and you get the full attention matrix, lower-triangular because of the mask: how much each token attends to itself and everything before it.

A single attention head has to average over everything it looks at, which the paper's authors point out inhibits capturing more than one kind of relationship at a time. Running several heads in parallel, each with its own learned projections, lets the model jointly attend to different representation subspaces at once.

Each head has its own learned Query, Key, and Value matrices, so each one can end up focused on something different: nearby words, subject-verb pairs, sentence position. The original paper uses 8 heads, each working in a 64-dimensional subspace of the model's 512 dimensions. The four below are simplified stand-ins so the different patterns are easy to see, shortened to "The cat sat mat" so a 4x4 grid fits cleanly; a real trained head rarely looks this clean.

The outputs of every head are concatenated back together and passed through one more learned matrix, W_O, mixing what each head found into a single vector per token. Nothing forces a head to specialize this way, it falls out of training because different patterns each help minimize the loss, and researchers have found this kind of specialization by inspecting real trained models.

Attention, as described so far, has no notion of distance. The causal mask already tells a token which positions came before it, but nothing about how far back, one step earlier and a hundred steps earlier look identical without more information.

So before attention runs, a positional encoding vector, built from sine and cosine waves at different frequencies, is added directly onto each token's embedding (first scaled up by sqrt(d_model), so its size is comparable to the positional signal being added to it).

The paper chose these specific sine and cosine waves over a learned position vector for two reasons: the fixed, regular pattern should make it easy for the model to learn to attend by relative position, and it should let the model handle sequences longer than any it saw during training.

In practice, this means every position in the sequence gets added a different, but consistent, fingerprint.

One pass of attention is not enough. The transformer stacks many identical blocks, each one refining every token's representation a bit further. The original paper stacks 6 on the decoder side; modern LLMs typically stack far more.

Inside each block, the attention output is added back to its input, a residual connection, then normalized with LayerNorm. This keeps gradients well-behaved as more blocks stack up.

The residual connection matters because it gives gradients a direct, unobstructed path backward: instead of a block having to learn the whole next representation from scratch, it only has to learn a correction, an "add this on top" delta, on top of an identity path that always passes the input straight through. That shortcut is what makes stacking dozens of blocks trainable at all.

LayerNorm does something more literal: for each token's vector on its own, it subtracts that vector's mean and divides by its standard deviation, forcing every token into the same predictable scale (mean 0, variance 1) before it enters the next block, then applies a small learned rescale. Watch the values below: same shape, same relative pattern, just rescaled onto common ground.

After that, a feedforward network processes each token's vector independently, expanding it to a wider hidden size and back down (512 to 2048 and back, in the original paper) before the next block begins.

After the last block, the final token's vector has absorbed information from the whole sequence, filtered through every layer of attention and feedforward processing.

That vector is projected back to the size of the vocabulary: one score, a logit, for every token the model knows about. The paper reuses the embedding table itself as this projection, rather than learning a separate matrix, so the same weights map a token to a vector and a vector back to token scores.

Softmax turns those scores into the same kind of probability distribution we started with, this time computed from the model's real internal state instead of illustration.

Every weight matrix used above, the embedding table, W_Q, W_K, W_V, W_O, the feedforward weights, starts out random.

Training feeds in real text, compares the predicted next-token distribution to the actual next token with a loss function, and measures how wrong the guess was.

Backpropagation computes how much each individual weight contributed to that error, by running the error signal back through the network, the reverse of the normal forward propagation direction. Gradient descent then nudges every weight slightly in the direction that would have made the correct answer more likely.

Repeat that across billions of examples, and the random matrices we started with become the ones that produced every visualization on this page. The full mechanics of backpropagation are their own topic, for a future explainer.