Transformer Architecture

2.1 Why open the box

You can build a working AI product knowing only Chapter 1. Plenty of people do. But three questions keep coming back, and none of them can be answered from outside the box.

• Why did a 4K context window cost so much more than 2K, rather than twice as much?
• Why can the model read a long prompt quickly but write a long answer slowly?
• Why did everything in AI suddenly accelerate after 2017?

All three have the same answer: attention. This chapter builds that mechanism from nothing, with no mathematics beyond multiplication and addition.

2.2 The problem: order carries meaning

Start with the actual difficulty. Language is a sequence, and the same words in a different order mean different things.

"the jacket is under the coat"
"the coat is under the jacket"

"return the item I bought"
"I bought the item, return"

A system that treats a sentence as an unordered bag of words cannot tell these apart. That was a genuine limitation of the statistical methods described in section 1.5 — they counted words without modelling arrangement.

There is a second, harder problem. Meaning depends on relationships between distant words, not just neighbouring ones.

2.3 What came before: recurrent networks

The pre-2017 answer was the recurrent neural network, or RNN, an idea dating to the 1980s and made practical by the LSTM variant in 1997.

The intuition is close to how a person reads: left to right, carrying a running sense of the sentence. That is genuinely appealing, and it worked well enough to power Google Translate from 2016.

2.4 The two walls RNNs hit

Recurrent networks were the state of the art for years. They were abandoned almost overnight because of two limitations that no amount of engineering could remove. Both are visible in Figure 2.1.

Wall one: information decay over distance

All memory of the sentence so far lives in one fixed-size bundle of numbers. Every new word overwrites part of it. By the time the model reaches "they", the trace of "shoes" has survived four rewrites and been diluted each time.

The LSTM, introduced in 1997, was designed specifically to slow this decay with learned gates controlling what to keep and what to discard. It genuinely helped, and it is why RNNs remained viable into the 2010s. But it delayed the problem rather than removing it. Beyond roughly 50 to 100 words, early context still faded.

Wall two: training could not be parallelised

This is the one that actually killed the architecture, and it is an engineering constraint rather than a quality one.

Step 4 cannot begin until step 3 has produced its output. That dependency is inherent to the design. A GPU has thousands of cores that can work simultaneously, and a strictly sequential algorithm leaves nearly all of them idle.

2.5 Attention, invented as a patch

Attention did not arrive as a replacement for recurrent networks. It arrived in 2014 as a fix for one of their symptoms, and only later turned out to be strong enough to stand alone.

The intuition is worth sitting with, because it is the whole idea. When a human translates "the shoes arrived scuffed" into another language, they do not memorise the sentence and then recite. They keep the sentence in front of them, and while writing each word they glance at the parts that matter for that word.

Attention is that glance, made mathematical: a weighted lookup where the model decides how much each other word matters right now.

Two properties fall out of this immediately, and they are exactly the two walls:

• Distance stops mattering. Scoring "them" against "shoes" is one operation regardless of whether they are 3 words or 300 apart. No information has to survive intermediate rewrites.
• Every word can be scored simultaneously. The score between "them" and "shoes" does not depend on the score between "them" and "Tuesday", so all of them can be computed at once, in parallel, on a GPU.

For three years attention was an add-on: recurrent networks with an attention mechanism bolted alongside. Then someone asked the obvious question.

2.6 2017: attention is all you need

If attention already lets a model look directly at every word, what is the recurrence actually contributing? In June 2017, eight researchers at Google published a paper whose title answered that: Attention Is All You Need.

2.7 Query, key and value

This is the heart of the chapter. Three terms that sound abstract until you see where the names come from, at which point they become obvious.

The names are borrowed from databases. Think about looking something up in a dictionary, or a product lookup in a shop system:

A normal database lookup is exact: the key either matches or it does not, and you get back one item. Attention is a soft lookup: every key matches to some degree, and you get back a blend of all the values, weighted by how well each key matched.

Now the part that trips people up. In self-attention, every word produces all three. Each word simultaneously asks a question, advertises what it offers, and carries content to contribute.

2.8 Attention scores, step by step

Time to actually compute one. We will use the sentence fragment "the blue shoes arrived scuffed" and work out what the word "scuffed" pays attention to. The numbers are illustrative but the procedure is exactly what runs inside every model.

1. Produce a query, key and value for every word Five words in, so five queries, five keys and five values. Each is a list of numbers. We are computing the output for "scuffed", so we take its query and score it against every key, including its own.
2. Score the query against each key with a dot product A dot product multiplies two vectors element by element and sums the result. It is large when two vectors point in a similar direction and small when they do not. That single number is how well this key answers this query.
3. Scale the scores down Divide each score by the square root of the vector length. Without this, long vectors produce large dot products, and large numbers make the next step behave badly — probability collapses onto a single word and learning stalls. This is why the technique is called scaled dot-product attention.
4. Softmax into weights Convert the scaled scores into positive numbers that sum to 1, exactly as in section 1.12. These are the attention weights: how much of each word to use.
5. Blend the values Multiply each word's value vector by its weight and add them all up. The result replaces the representation of "scuffed" — now enriched with context from the words that mattered.

import numpy as np

def attention(Q, K, V):
    scores  = Q @ K.T # 1. every query against every key
    scores  = scores / np.sqrt(K.shape[-1]) # 2. scale
    weights = softmax(scores, axis=-1) # 3. to probabilities
    return weights @ V # 4. weighted blend of values

That is genuinely the whole thing. Every language model you have used is this operation, repeated across many heads and many layers, with a few supporting pieces covered in section 2.11.

2.9 Multi-head attention

One attention operation produces one set of weights, which means it can express one kind of relationship at a time. That is a real limitation, because words relate to each other in several ways simultaneously.

Forcing all four through a single set of attention weights means compromising on all of them. The solution is straightforward: run several attention operations in parallel, each with its own learned query, key and value matrices.

The detail people miss

Adding 8 heads does not make the model 8 times more expensive. The vector is divided among the heads rather than duplicated. If a model works with 512-number vectors and has 8 heads, each head operates on 64 numbers. Total work is roughly unchanged; it has simply been split into 8 specialised views instead of one general one.

2.10 Positional encoding

Section 2.6 flagged a debt: removing recurrence threw away word order. Time to pay it.

Attention as described so far is order-blind. It computes scores between pairs of words, and nothing in that computation refers to where a word sits. Shuffle the input and you get the same set of attention scores, just rearranged.

Approaches, and why the obvious ones fail

2.11 The rest of the block

Attention is the interesting part, but it is not the whole layer. A transformer block has three supporting components, and each exists to solve a specific problem.

Feed-forward network: where thinking happens

Attention moves information between words. It does not do much processing of the information itself — it is fundamentally a weighted average. After attention, each word passes through a small neural network on its own.

A useful division of labour: attention decides what is relevant; the feed-forward network decides what that means.

Residual connections: keeping deep networks trainable

Modern models stack many layers — 32, 80, sometimes more. Stacking naively fails, because the signal used to adjust weights during training shrinks as it passes back through each layer, until the earliest layers stop learning altogether.

Layer normalisation: keeping numbers in range

With many layers each transforming numbers, values can drift steadily larger or smaller until training becomes unstable. Layer normalisation rescales each vector to a consistent statistical range before it enters the next sub-layer. It is plumbing rather than intelligence, but without it deep transformers do not train reliably.

2.12 Encoder, decoder, or both

The original 2017 transformer had two halves: an encoder that read the source sentence and a decoder that wrote the translation. That made sense for translation. It turned out you can use either half on its own, and which half you keep determines what the model is good at.

2.13 Causal attention

There is a problem with training a generative model. If the model can see the entire text at once, then when learning to predict word 5 it can simply look at word 5. It would score perfectly during training and produce nonsense in use, because at generation time the future does not exist yet.

The payoff is subtle and important. Because every position is masked independently, the model can be trained on an entire document in one parallel pass while still learning to predict each token from only its predecessors. A 1,000-token document provides 1,000 separate prediction exercises, all computed simultaneously.

2.14 Why context costs what it does

Now the question from section 2.1 can be answered properly.

Attention scores every token against every other token. With 10 tokens that is 100 comparisons. With 1,000 tokens it is 1,000,000. Double the input and the work quadruples, because you are filling a square grid.

How the industry escaped the squeeze

Modern models handle 128K or more, which the table suggests should be impossible. Three developments changed the picture.

2.15 Why transformers won

It is worth being precise about this, because the popular explanation is wrong. Transformers did not win because attention is a uniquely brilliant model of language. They won because of hardware.

That unlocked the sequence everything else followed from: train bigger models, on more data, in reasonable time. Which produced the scaling behaviour of 2020 onward, which produced the capabilities that made products like chat assistants viable.

2.16 Transformer limitations

Transformers are dominant, not perfect. Knowing the sharp edges is what separates understanding from repetition.

2.17 Key takeaways

2.18 Interview drills

Transformer questions separate people who have read a blog post from people who understand the mechanism. The tell is usually whether you can explain why a design choice exists, not just name it.

Expand all

2.19 Where this leads

The box is open. You know what happens between prompt and response, and you can explain the cost and latency behaviour of any model you use from first principles.

One thing was assumed throughout this chapter and never justified: that words can be turned into vectors where mathematical operations correspond to meaning. Attention compares query and key vectors with a dot product and treats a high value as "relevant" — but why should arithmetic on lists of numbers have anything to do with meaning at all?

Chapter 3 answers that. Embeddings are the bridge between language and mathematics, and they are also the foundation of semantic search, which is the first genuinely useful AI feature most retail teams ship.