Embeddings and Semantic Meaning

3.1 The debt from Chapter 2

Chapter 2 described attention comparing a query vector against a key vector with a dot product, and treating a large result as "these two words are relevant to each other". That was smuggled past you.

Why should arithmetic on a list of numbers have anything to do with meaning? Nothing in multiplication knows that "shoes" and "footwear" are related. If the numbers were assigned arbitrarily, the dot product would be meaningless and attention would be noise.

So the numbers cannot be arbitrary. They have to be arranged so that geometric closeness corresponds to similarity of meaning. That arrangement is what an embedding is, and this chapter is about how it is produced.

3.2 What a vector is

No jargon without explanation, so: a vector is an ordered list of numbers. That is the entire definition.

[2, 5]              a 2-dimensional vector
[2, 5, 9]           a 3-dimensional vector
[0.21, -0.88, ...]  a 1536-dimensional vector (typical for embeddings)

The useful part is that a list of numbers can be read as coordinates. The vector [2, 5] is a point 2 across and 5 up. And once things are points in a space, "how similar are they" becomes "how close together are they" — a question arithmetic can answer.

winter parka     [180, 9]
fleece jacket    [45, 6]
windbreaker      [40, 2]
cotton t-shirt   [12, 1]

3.3 First attempt: one-hot encoding

Before the clever solution, the obvious one — because understanding why it fails makes everything after it feel inevitable.

You need to turn words into numbers. The simplest scheme: number every word in your vocabulary, then represent each word as a list that is all zeros except a single 1 in that word's slot.

          shoes  boots  jacket  coat  laptop
shoes  =  [  1  ,  0  ,   0   ,  0  ,   0   ]
boots  =  [  0  ,  1  ,   0   ,  0  ,   0   ]
jacket =  [  0  ,  0  ,   1   ,  0  ,   0   ]
coat   =  [  0  ,  0  ,   0   ,  1  ,   0   ]
laptop =  [  0  ,  0  ,   0   ,  0  ,   1   ]

Why it fails, and the failure is total

Take the dot product of "shoes" and "boots": multiply position by position and sum. (1×0) + (0×1) + 0 + 0 + 0 = 0.

Now "shoes" and "laptop": also 0.

There is a second problem, which is practical rather than conceptual.

Two megabytes to say "this is product number 40,127". The information content is one number; the storage is half a million.

3.4 The distributional hypothesis

One-hot fails because we assigned the numbers by hand, arbitrarily. To do better, the numbers have to be learned from something. But learned from what? Meaning is not written down anywhere.

The answer is one of the most productive ideas in the history of language technology, and it predates all of this by decades.

This is worth pausing on, because it converts an impossible problem into a countable one. You cannot look up what "trainers" means. You can count what words appear near it across millions of documents.

...comfortable TRAINERS for running, laces, size 9, cushioned...
...these SNEAKERS are comfortable, good laces, true to size 9...
...wore my TRAINERS running daily, cushioned sole, great fit...
...SNEAKERS with cushioned soles, perfect for running, true fit...

3.5 Counting words: TF-IDF and its limits

The distributional hypothesis says to look at context. The first serious attempt to operationalise that was simply to count, and the resulting technique is still in production use today — which is why it deserves proper treatment rather than a dismissal.

Multiply the two and you get a useful signal. In a product catalogue, "the" appears in every listing and scores near zero. "Gore-Tex" appears in a handful and scores highly for those — it is genuinely distinguishing.

Why it is not enough

TF-IDF still represents documents by which words they contain. It weights those words more intelligently than one-hot, but the underlying comparison is still exact word overlap.

3.6 2013: Word2Vec

Counting contexts gives you enormous, sparse vectors. The breakthrough was realising you could learn small dense ones instead — and that a neural network would do it for you if you gave it the right task.

This is a pattern worth recognising, because it recurs throughout AI: train on a task you do not care about, to obtain a representation you do care about. It is the same logic as next-token prediction in Chapter 1 producing a model that can write emails.

The two training arrangements

3.7 Arithmetic on meaning

Word2Vec produced a result that startled people, including its authors. The embedding space turned out to have directions that correspond to concepts, which means you can do arithmetic with meaning.

vector("king") - vector("man") + vector("woman")  ~=  vector("queen")

Read that as a set of directions. Subtracting "man" and adding "woman" moves you along a consistent gender direction in the space. Starting at "king" and taking that step lands you very near "queen". The same direction connects "actor" to "actress" and "uncle" to "aunt".

Other directions encode other relationships: a plural direction, a past-tense direction, a country-to-capital direction. None of these were designed. They emerged from predicting neighbouring words.

"trainers"  - "casual"  + "formal"    ->  "dress shoes"
"t-shirt"   - "summer"  + "winter"    ->  "jumper"
"iPhone"    - "Apple"   + "Samsung"   ->  "Galaxy"

3.8 Similarity metrics

We have been saying "similar" and "near" for six sections without defining either. Two vectors are lists of numbers; turning that into a single similarity score requires choosing a formula, and the three common choices disagree with each other in ways that matter.

Why length is the whole story

The difference between these metrics comes down to one question: should the magnitude of a vector affect similarity? In text embeddings, magnitude tends to encode things you usually do not care about — how long the document was, how frequently its words occurred, how emphatic the phrasing was.

Picture a short product title and a long product description for the same jacket. They point in nearly the same direction, because they are about the same thing, but the longer text often produces a longer vector. Cosine similarity says they are nearly identical, which is what you want. Euclidean distance says they are far apart, because one point sits much further from the origin than the other.

"""Where cosine and Euclidean disagree, demonstrated rather than asserted.

The claim in the prose is that magnitude encodes document length, and that
Euclidean distance is fooled by it while cosine is not. That is easy to say
and easy to get backwards, so the numbers here are computed from vectors
chosen to make the disagreement visible.

No numpy: the arithmetic is short enough to write out, and seeing the
formulas spelled out is the point of the exercise.
"""
from __future__ import annotations

from math import sqrt


def dot(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


def norm(a: list[float]) -> float:
    return sqrt(dot(a, a))


def cosine(a: list[float], b: list[float]) -> float:
    """Angle only: divides out both magnitudes."""
    return dot(a, b) / (norm(a) * norm(b))


def euclidean(a: list[float], b: list[float]) -> float:
    """Straight-line distance. Smaller means closer."""
    return sqrt(sum((x - y) * (x - y) for x, y in zip(a, b)))


def normalise(a: list[float]) -> list[float]:
    length = norm(a)
    return [x / length for x in a]


# A short title and a long description of the SAME jacket point in nearly the
# same direction, but the longer text produces a longer vector.
TITLE = [0.6, 0.8, 0.1]
DESCRIPTION = [3.0, 4.0, 0.5]        # ~5x longer, same direction

# A different product that happens to sit at a similar distance from the origin
# as the title does.
UMBRELLA = [0.1, 0.2, 0.9]

print("same jacket, different text lengths")
print(f"  cosine     {cosine(TITLE, DESCRIPTION):.3f}   (1.0 = identical direction)")
print(f"  euclidean  {euclidean(TITLE, DESCRIPTION):.3f}   (0.0 = identical point)")

print("\ndifferent products, similar vector lengths")
print(f"  cosine     {cosine(TITLE, UMBRELLA):.3f}")
print(f"  euclidean  {euclidean(TITLE, UMBRELLA):.3f}")

print("\nwhich does each metric call the better match for the title?")
cos_pick = "description" if cosine(TITLE, DESCRIPTION) > cosine(TITLE, UMBRELLA) else "umbrella"
euc_pick = "description" if euclidean(TITLE, DESCRIPTION) < euclidean(TITLE, UMBRELLA) else "umbrella"
print(f"  cosine says     -> {cos_pick}")
print(f"  euclidean says  -> {euc_pick}")

# After normalising, the disagreement disappears entirely.
t, d = normalise(TITLE), normalise(DESCRIPTION)
print("\nafter normalising both to unit length")
print(f"  cosine     {cosine(t, d):.3f}")
print(f"  euclidean  {euclidean(t, d):.3f}")
print(f"  dot product {dot(t, d):.3f}  <- now identical to cosine")

same jacket, different text lengths
  cosine     1.000   (1.0 = identical direction)
  euclidean  4.020   (0.0 = identical point)

different products, similar vector lengths
  cosine     0.333
  euclidean  1.118

which does each metric call the better match for the title?
  cosine says     -> description
  euclidean says  -> umbrella

after normalising both to unit length
  cosine     1.000
  euclidean  0.000
  dot product 1.000  <- now identical to cosine

The two metrics choose opposite answers. Cosine sees that the title and the description point in exactly the same direction and scores a perfect 1.0. Euclidean sees a gap of 4.2 between them — larger than the 1.118 separating the jacket from an umbrella — and concludes the umbrella is the better match. It is not confused about meaning; it is faithfully reporting that one point sits far from the other, which here is entirely an artefact of description length.

The final block is the practical resolution. Once both vectors are scaled to unit length, cosine reads 1.000, Euclidean reads 0.000, and the dot product equals cosine exactly. All three now agree, because the only thing they could have disagreed about has been removed.

3.9 Why averaging word vectors fails

Word2Vec gives you a vector per word. Search needs a vector per sentence or per product description. The obvious bridge is to average the word vectors together, and for a while that is exactly what everyone did.

It works better than it has any right to. Average the vectors for "waterproof", "hiking" and "jacket" and you land somewhere genuinely sensible — near other outdoor clothing. As a cheap baseline it is respectable, and it was the standard approach for several years.

It also fails in three specific ways, and each failure points directly at what had to be invented next.

1. Word order disappears entirely Averaging is commutative, so dog bites man and man bites dog produce the identical vector. For retail, jacket for dog and dog jacket are the same query, which is fine, but case for phone and phone for case are not. Any meaning carried by structure is lost before you start.
2. Common words drown out rare ones In a case for the new iPhone, the words a, for and the contribute three vectors of generic filler against one meaningful iPhone. The average drifts toward the centre of the space, where all bland text lives. Weighting by TF-IDF from 3.5 helps considerably, which is why that combination became the standard trick, but it is a patch rather than a fix.
3. One vector per word, regardless of meaning This is the fundamental one. Word2Vec assigns bank exactly one vector, which must simultaneously serve the riverbank and the financial institution. The result is a compromise vector sitting between two unrelated meanings, representing neither. Every word with more than one sense is permanently blurred.

3.10 Contextual embeddings

The fix for polysemy is to stop storing a vector for each word and start computing one each time the word appears, using the sentence around it.

The change is best seen in a single word appearing twice:

"I sat on the river bank"        ->  bank = [ 0.21  -0.88   0.43  ... ]
"I deposited it at the bank"     ->  bank = [-0.72   0.15   0.66  ... ]

cosine(these two) = 0.18       # correctly recognised as different senses

"I withdrew cash from the bank"  ->  bank = [-0.69   0.19   0.71  ... ]

cosine(financial pair) = 0.96  # correctly recognised as the same sense

Nothing about the token bank changed. What changed is that the model read the surrounding words first — and the attention mechanism from Chapter 2 is precisely the machinery that lets river or deposited reach across the sentence and reshape what bank means here.

3.11 Sentence-BERT and bi-encoders

BERT understood language but could not compare sentences. The fix, in 2019, was not a new architecture — it was a new training objective, and it is the reason semantic search became practical.

The problem in numbers

Before Sentence-BERT, the accurate way to compare two sentences with BERT was to feed them in together and let attention run across both, producing a similarity score. This works well and is the cross-encoder design. It also does not scale, and the arithmetic is worth doing.

How Sentence-BERT actually trains

Reimers and Gurevych's insight was that if you want vectors whose distances are meaningful, you must train on distances. So they fine-tuned BERT on pairs of sentences with a known relationship, using a structure that forces the geometry into shape.

1. Run two texts through the same model A siamese arrangement: one set of weights, applied separately to each text. Identical inputs must give identical outputs, so the two are directly comparable.
2. Pool the token vectors into one Mean pooling over the token outputs, which turns out to beat using the [CLS] token. This is the same averaging that failed in 3.9, but it now works — because the next step trains the model so that it does.
3. Train on a distance objective Show the model a sentence, something that means the same, and something that does not. Penalise it when the matching pair is not closer than the mismatched pair. This is contrastive learning, and it directly optimises the property you intend to use at query time.

3.12 Choosing a model, and what MTEB does not tell you

There are hundreds of embedding models and a public leaderboard ranking them. Picking the top row is the obvious move and usually the wrong one.

The dimensions that actually decide it

3.13 Chunking strategies

An embedding model turns a piece of text into one vector. That forces a question with no universally right answer: how big should the piece be?

The constraint is dilution. One vector holds a fixed amount of information, so the more text you push into it, the more each individual idea gets averaged away. This is the same failure as 3.9, operating at document scale rather than word scale.

3.14 Multilingual embeddings

If your catalogue serves several markets, you have a choice: one index per language, or one space holding all of them. The second option relies on a genuinely surprising property.

The practical consequence is worth stating plainly, because it sounds too good to be true. A customer typing a Spanish query can match an English product description that was never translated. You search across every market's content with one query, in one index, with no translation step anywhere in the pipeline.

How the alignment is trained

The usual technique is knowledge distillation, and Reimers and Gurevych described a clean version of it in 2020. Take a strong English-only model as the teacher. Then train a multilingual student so that, given a translated pair, it produces the same vector the teacher produced for the English side. The student learns to map every language into the teacher's existing English space, which is what makes the languages line up.

3.15 Images and joint spaces

Everything so far has been text. The same idea works for images, and the interesting part is what happens when both live in one space.

What this unlocks in a catalogue

3.16 Fine-tuning on your catalogue

A general-purpose embedding model learned the internet's idea of similarity. Your business has its own, and the gap between the two is where fine-tuning pays.

A public model knows that laptop and notebook computer are similar. It does not know that in your catalogue SKU-88421 and Trailmaster II are the same product, that customers searching school shoes want black leather rather than trainers, or that your ONE+ range is a battery platform rather than a model name. None of that exists on the internet in a form a model could have learned it from.

Where the training pairs come from

This is the part people assume will block them, and usually does not. Most retailers are already sitting on the labels.

1. Click and purchase logs A query, and the product the customer then clicked or bought, is a positive pair. This is the best source by a wide margin: real user intent, arriving continuously, with nobody labelling anything. It is also the most honest signal you have about what your customers think similar means.
2. Products the customer skipped Items shown high in the results and not clicked are hard negatives — superficially relevant but wrong. They teach far more than random negatives, because the model already knows a drill is not a dress. What it needs is the distinction between two similar drills.
3. Your own catalogue structure Products sharing a category, brand or specification are weakly positive pairs. Cheap to generate in bulk from data you already maintain.
4. Synonym and correction lists Whatever your search team has accumulated: spelling corrections, regional terms, the fact that jumper and sweater are the same garment. Small, high quality, and usually already written down.

3.17 Evaluating embedding quality

Every recommendation in this chapter has ended with "measure it on your own data". This section is how.

Building the evaluation set

1. Take real queries, not invented ones Pull the top few hundred from your search logs, and deliberately include the long tail as well as the head. Queries written by your team are systematically too clean: they spell things correctly and use internal vocabulary. Real ones contain typos, partial product names and regional terms, which is exactly where models differ.
2. Label what should come back For each query, list the products a knowledgeable person would expect. You do not need every relevant item — a handful of clear positives per query is enough to compute recall meaningfully.
3. Include the awkward cases on purpose Exact product codes, misspellings, questions phrased as sentences, queries with no good answer, and near-duplicate products that must be told apart. Averages hide these, and they are where real systems fail.
4. Score per category, never as one average Report recall separately for each of those groups. A model can improve overall while collapsing on product codes — a small share of queries carrying a large share of purchase intent.

3.18 Key takeaways

3.19 Interview drills

Embeddings are where interviewers check whether you understand the machinery or have only used the API. The strong answers below all do the same thing: explain a mechanism, then name its limit.

Expand all

Where this leaves you

Chapter 2 asked you to accept that meaning can live in a list of numbers. That debt is now paid. You know where those numbers come from, why the space has usable structure, how to compare two positions in it, and why the model that produced them is good at exactly the task it was trained on and no other.

You can also make the practical decisions: which model to use and why the leaderboard cannot answer that for you, how to cut text so each vector is about one thing, when a multilingual or joint image space earns its cost, and how to prove any of it on your own catalogue rather than on someone else's benchmark.

What you cannot yet do is store 800 million of these vectors and search them in milliseconds. Comparing a query against every vector works fine for a thousand products and is hopeless at catalogue scale. Chapter 4 is about the data structures and engineering that make it fast: approximate nearest neighbour indexes, quantization, sharding, and the arithmetic for sizing a cluster you can actually afford.