Retrieval-Augmented Generation

5.1 The problem RAG solves

A language model is a fixed snapshot of whatever it read during training. That single fact causes four distinct problems, and RAG exists to address all four at once.

Suppose a customer asks your storefront assistant: "Is the Dyson V15 still covered by the extended warranty promotion, and does it ship to Chile?"

A bare language model will answer confidently. It will also be wrong, because it cannot possibly know. Here is precisely why.

What people tried before RAG

5.2 What RAG actually is

The three letters, in order

1. Retrieval — find candidate evidence The user's question is used to search a knowledge store. This is ordinary information retrieval: vector search, keyword search, or both. Output is a handful of text passages that might be relevant. Nothing intelligent has happened yet.
2. Augmentation — build the prompt The retrieved passages are assembled into a prompt alongside the question and an instruction such as "answer using only the context below". This step is pure string construction, and it is where more RAG systems fail than anyone expects.
3. Generation — produce a grounded answer The model reads the assembled prompt and writes an answer. Because the facts are in front of it, it does not need to recall them — it needs to read them. Reading is a much easier task than remembering, and models are far more reliable at it.

5.3 RAG versus fine-tuning versus long context

Three ways to make a model produce answers it could not produce out of the box. They are frequently presented as competitors. They are not — they solve different problems, and the interview question is really testing whether you know which.

The long-context argument

Context windows grew from 4,000 tokens in 2022 to a million or more by 2024, and this produced a genuinely reasonable question: if you can paste the entire manual into the prompt, why bother retrieving?

Four reasons, in order of how often they actually bite.

1. Cost scales with every single query A million-token prompt costs a thousand times a thousand-token prompt. At any real query volume this dominates your bill. Retrieval means paying for 2,000 tokens of relevant context instead of 900,000 tokens of mostly-irrelevant context.
2. Latency scales too Processing a very long prompt takes seconds before the first token appears. A customer-facing assistant cannot spend eight seconds reading a manual it did not need.
3. Accuracy degrades in the middle The well-documented "lost in the middle" effect: models attend reliably to the start and end of a long text and much less reliably to the middle. Burying the answer at 40% depth in a huge prompt measurably reduces the chance it is used.
4. Your corpus is bigger than any window This is the decisive one. A million tokens is roughly 4 MB of text. A retail catalogue with policies, specifications and reviews is measured in gigabytes. No context window will ever hold it, and the gap grows faster than window sizes do.

5.4 Anatomy of a RAG system

Before going deep on any single stage, here is the whole machine. Two pipelines run at completely different times, and confusing them is a common source of muddled thinking.

5.5 Loading and parsing

The least glamorous stage, and the one that quietly determines your ceiling. No amount of clever retrieval recovers from text that was mangled on the way in.

What each format costs you

Capacity Spin Speed Energy 9kg 1400rpm A+++ 8kg 1200rpm A++

Cleaning: what to strip and what to keep

5.6 Chunking

Chunking is the decision that most affects RAG quality, and the one teams spend least time on. It is worth understanding exactly why it is necessary before choosing a strategy.

Why chunk at all?

Three independent reasons, each sufficient on its own.

1. Embedding models have input limits Most accept 512 tokens; some reach 8,000. A 200-page manual simply does not fit.
2. One vector cannot represent a long document This is the deeper reason. Averaging a whole manual into 768 numbers produces a vector that means "washing machine manual, generally" — equally mediocre for every specific question. Meaning gets diluted. Smaller chunks give sharper vectors.
3. The context window is a budget You retrieve several passages. If each is 10,000 tokens you can afford one; if each is 400 you can afford twelve and let reranking pick the best.

The five strategies, weakest to strongest

separators = ["\n\n", "\n", ". ", " ", ""]
             para   line  sent  word  char

Overlap

Overlap means consecutive chunks share some text at their boundary. It exists to solve one specific problem.

CHUNK 1: "...Electronics purchased during a promotional period are
          subject to modified return terms."
CHUNK 2: "The window is 14 days rather than the standard 30."

5.7 Metadata extraction

Every chunk should carry structured fields alongside its text. This is cheap at ingestion and impossible to add later without re-processing everything.

RAW CHUNK
  "The window is 14 days rather than the standard 30."

WITH CONTEXTUAL HEADER
  "Returns Policy > Mexico > Promotional Purchases > Electronics
   The window is 14 days rather than the standard 30."

5.8 Embedding and incremental ingestion

Embedding is mechanically simple — batch your chunks through the model. The engineering is in everything around it.

Incremental ingestion

Re-embedding an entire corpus nightly is the default naive design and is almost always wasteful. The fix is the same content-hash idea used in chapter 4.

1. Hash the chunk text at ingestion Store the hash in the payload alongside the vector.
2. On the next run, re-chunk and re-hash Compare against what is indexed. Three outcomes: unchanged, changed, or gone.
3. Act on the difference only Unchanged chunks are skipped entirely — no embedding call. Changed chunks are re-embedded and upserted. Deleted chunks are removed.

5.9 Dense retrieval and its blind spots

Dense retrieval is what chapter 4 built: embed the query, find the nearest chunk vectors. It is the default for RAG and it is genuinely good. It also fails in specific, predictable ways that you must know to design around.

5.10 Sparse retrieval and BM25

5.11 Hybrid retrieval and fusion

Dense finds meaning. Sparse finds exact terms. Their failures are largely complementary, which is the entire argument for running both.

The score-combination trap

The obvious approach is to add the scores together, perhaps with a weight. It does not work, and understanding why is the point of this section.

score(doc) = sum over each retriever of  1 / (k + rank)

with k = 60 by convention

5.12 Query transformation

Everything so far has assumed the user's question is a good search query. It usually is not. Query transformation is the practice of rewriting the question before it ever reaches the retriever.

Four transformations worth knowing

5.13 Top-k and similarity thresholds

Two small numbers with outsized influence: how many chunks you retrieve, and how similar something must be before you keep it at all.

Choosing k

The instinct is that more context is better. It is not. Three forces push in opposite directions.

1. Too small and you miss the answer If the answer lives in the chunk ranked 6th and k is 3, no amount of prompting saves you. The information is simply absent.
2. Too large and you dilute the signal Irrelevant chunks are not neutral filler. They actively distract the generator, which will sometimes answer from a plausible-looking wrong passage.
3. Every chunk costs tokens and time Cost and latency scale with context length, and long contexts degrade the positional attention discussed in 5.17.

Similarity thresholds, and why they disappoint

Two approaches survive contact with production. Relative thresholds keep results within some fraction of the top score, adapting per query rather than fixing a global constant. Reranker scores are the better answer where you can afford one: a cross-encoder produces a calibrated relevance judgement, so a cutoff on its score means roughly the same thing from query to query. That calibration is a large part of why rerankers are worth their cost.

5.14 Parent-child retrieval

The chunking dilemma from 5.6 had no good answer: small chunks embed precisely but lack context, large chunks carry context but embed vaguely. Parent-child retrieval refuses the trade entirely.

Three ways to build it

5.15 Reranking

The highest-leverage component in most RAG systems, and the one most often missing. If you add a single thing to a naive pipeline, add this.

5.16 Context compression

Even after reranking, retrieved passages carry a great deal of text that has nothing to do with the question. Compression strips it before it reaches the generator.

A chunk that earns its place in the top 5 might be 800 tokens of which perhaps 60 actually bear on the question. The rest is boilerplate, neighbouring subject matter and legal throat-clearing. That surplus costs money, adds latency, and — as 5.17 covers — pushes the useful sentence toward the middle of the context where models attend to it least.

5.17 Context construction and order

You have five excellent chunks. The order in which you paste them into the prompt changes the answer. This surprises almost everyone the first time they measure it.

Every stage so far has been about choosing text. This stage is about arranging it. It is tempting to assume that once the right passage is inside the context window the model will find it, because the whole window is visible to attention at once. That assumption is wrong, and the way in which it is wrong has a name.

Why the U-shape exists

Two forces combine. Text near the start is privileged because attention patterns learned during training treat opening tokens as scene-setting; some attention heads latch onto the first few tokens almost unconditionally. Text near the end is privileged because it sits closest to the position where generation begins, and because autoregressive training makes recent tokens the strongest predictor of the next one. The middle enjoys neither advantage.

def reorder_for_attention(chunks):
    """Place the strongest chunks at the start and end of the prompt.

    `chunks` arrives sorted best-first from the reranker. Models attend most
    to the beginning and the end of a context window (Liu et al., 2023), so
    feeding them in plain rank order buries rank 2 in the weakest position.
    """
    head, tail = [], []
    for position, chunk in enumerate(chunks):
        # Even ranks build the opening run; odd ranks build the closing run,
        # which is reversed at the end so rank 2 finishes the prompt.
        (head if position % 2 == 0 else tail).append(chunk)
    return head + list(reversed(tail))


# ranked[0] is the best chunk the reranker found.
ranked = ["c1", "c2", "c3", "c4", "c5"]
print(reorder_for_attention(ranked))
# ['c1', 'c3', 'c5', 'c4', 'c2']  -- best leads, second best closes

What else goes in the context, and where

Order matters beyond the chunks themselves. A retrieval-augmented prompt has five parts, and each has a position that works better than the alternatives.

5.18 The grounding prompt

The instructions that tell the model to answer from the documents and nothing else. This is where retrieval turns into a trustworthy answer, or fails to.

A model given documents will happily blend them with what it already believes. Ask about a return window and it may average your policy against the thousands of return policies it saw in pre-training. The grounding prompt exists to forbid that blending, and the wording is load-bearing.

Building the prompt one rule at a time

Rather than presenting a finished template to copy, it is worth watching it grow. Each rule below was added because something went wrong without it.

1. Start with the role and the hard constraint You are a customer service assistant. Answer using only the documents provided below. The word only is doing real work. Without it, the model treats the documents as helpful background rather than as the boundary of what it may say.
2. Add the escape hatch If the documents do not contain the answer, say "I don't have that information" and stop. Without an explicit permitted failure, the model treats answering as mandatory and invents something. You are competing with a training signal that rewards helpfulness, so the refusal has to be made legitimate.
3. Add the citation requirement After each sentence, cite the document ID it came from, like [doc-3]. This is covered fully in 5.19. It also has a useful side effect: a model required to name its source is measurably less willing to fabricate, because every claim now needs somewhere to point.
4. Forbid the specific failure you are seeing Do not combine information from your own knowledge with the documents. Do not guess at policies not stated here. Generic instructions to be accurate achieve little. Naming the exact behaviour works far better.
5. Handle contradictions explicitly If two documents disagree, prefer the one with the more recent effective date and say that they disagree. Without this the model silently picks one, and you never learn that your corpus contains a conflict.

5.19 Citations and attribution

A cited answer can be checked. An uncited answer has to be trusted. In any setting where being wrong has a cost, that difference is the entire product.

Citation is often treated as a presentation detail bolted on at the end. It is better understood as the mechanism that makes the whole system auditable. When a customer disputes an answer, a cited response lets you open the exact paragraph the model used and settle the question in seconds. Without it you are debating what a model might have been thinking.

Three ways to produce citations

"""Verify that each cited quote genuinely appears in the chunk it cites.

Asking an LLM for citations is cheap; trusting them is not. This turns a
claimed citation into a checkable one: the model must quote the supporting
sentence, and we confirm that quote really exists in the chunk it named.

Exact string matching is too brittle -- models normalise whitespace, curly
quotes and casing. Token-overlap containment is tolerant of that reformatting
while still catching a quote that was invented outright.
"""
from __future__ import annotations

import re

# Below this fraction of quote tokens found in the source, we treat the
# citation as unsupported. 0.85 tolerates minor reformatting but rejects a
# sentence the model assembled itself.
SUPPORT_THRESHOLD = 0.85


def normalise(text: str) -> list[str]:
    """Lowercase and strip punctuation so formatting noise cannot fail a match."""
    return re.findall(r"[a-z0-9]+", text.lower())


def support_ratio(quote: str, source: str) -> float:
    """Fraction of the quote's tokens that appear in the source chunk."""
    quote_tokens = normalise(quote)
    if not quote_tokens:
        return 0.0
    source_tokens = set(normalise(source))
    hits = sum(1 for token in quote_tokens if token in source_tokens)
    return hits / len(quote_tokens)


def verify(claims: list[dict], chunks: dict[str, str]) -> list[dict]:
    """Attach a verdict to every claim the model made.

    Each claim is {"text": ..., "doc_id": ..., "quote": ...}. A claim fails if
    it cites a document that was never retrieved, or if its quote is not
    supported by that document's text.
    """
    results = []
    for claim in claims:
        source = chunks.get(claim["doc_id"])
        if source is None:
            verdict, ratio = "phantom-citation", 0.0
        else:
            ratio = support_ratio(claim["quote"], source)
            verdict = "supported" if ratio >= SUPPORT_THRESHOLD else "unsupported"
        results.append({**claim, "verdict": verdict, "support": round(ratio, 2)})
    return results


CHUNKS = {
    "doc-1": "Most items may be returned within 90 days of purchase with a receipt.",
    "doc-2": "Large electronics, including televisions, carry a 30-day return window.",
}

CLAIMS = [
    # Correct: quote lifted from the chunk it cites.
    {"text": "TVs must be returned within 30 days.",
     "doc_id": "doc-2",
     "quote": "televisions carry a 30-day return window"},
    # Right answer, wrong source -- the classic misattribution.
    {"text": "TVs must be returned within 30 days.",
     "doc_id": "doc-1",
     "quote": "televisions carry a 30-day return window"},
    # Fabricated policy that appears in no retrieved chunk at all.
    {"text": "Opened televisions cannot be returned.",
     "doc_id": "doc-2",
     "quote": "opened televisions are final sale"},
    # Cites a document that was never retrieved.
    {"text": "Returns are free for members.",
     "doc_id": "doc-9",
     "quote": "members return items free of charge"},
]

for row in verify(CLAIMS, CHUNKS):
    print(f"{row['verdict']:<18} support={row['support']:<5} {row['doc_id']}")

Running that produces one pass and three catches, which is the point of the exercise:

supported          support=1.0   doc-2
unsupported        support=0.14  doc-1
unsupported        support=0.2   doc-2
phantom-citation   support=0.0   doc-9

The second row is the interesting one. The claim is correct — televisions really do have a 30-day window — but it cites the 90-day general policy. A reader who clicked that citation would find text contradicting the sentence it supposedly supports. Only the verification step catches it, because the answer itself reads perfectly.

5.20 Abstention

Teaching the system to say "I don't know". The hardest behaviour to get right, and the one that decides whether anyone trusts the product after the first bad answer.

Every stage so far assumed retrieval found something useful. Sometimes it does not. The document may not exist, the question may be outside the corpus, or the user may be asking something no policy covers. The system needs a defined behaviour for that case, and the default behaviour — answer anyway, fluently — is the worst one available.

Where abstention can be triggered

There are four places to catch an unanswerable question, and a production system generally uses several. Each catches something the others miss.

1. Before generation, on retrieval scores If the best chunk after reranking scores below a threshold, abstain without calling the generator at all. Cheapest and fastest. Use the reranker's score, not the raw vector similarity — as 5.13 noted, cosine similarity has no absolute meaning, while a cross-encoder score is calibrated relevance.
2. Before generation, on the query itself Some questions should never reach retrieval: medical or legal advice, requests for another customer's data, or anything outside the assistant's scope. A cheap classifier on the incoming query handles these, and the refusal can be specific and helpful rather than generic.
3. During generation, by instruction The escape hatch from 5.18. Necessary but not sufficient, because it depends on the model choosing to comply when helpfulness pulls the other way.
4. After generation, by verification Run the citation check from 5.19 or a faithfulness judge from 5.33 over the draft. If the claims are not supported, discard the answer and abstain instead. Most expensive, and the only one that catches a fluent answer built on genuinely retrieved but irrelevant text.

A refusal is a user interface problem too

I don't have that information is technically correct and practically infuriating. A good abstention tells the user what the system does know and where to go next, which turns a dead end into a handover.

5.21 Naive, advanced, and modular RAG

Everything up to here has been components. This section is the map that shows how they assemble into three recognised generations of architecture, and why the field kept moving.

The naming comes from a 2023 survey by Gao and colleagues, Retrieval-Augmented Generation for Large Language Models, which grouped the rapidly multiplying techniques into three tiers. The labels stuck because they describe a real progression: each generation exists because the previous one hit a specific wall.

5.22 Routing and adaptive retrieval

Not every question needs a search. Some need three. Routing is the decision layer that looks at a query first and chooses what to do with it.

A naive pipeline retrieves unconditionally. Ask it hello and it dutifully embeds the greeting, searches 800 million products and pastes five irrelevant chunks into the prompt. Ask it a question spanning both the returns policy and live order status, and it performs exactly one vector search, because one search is all it knows how to do.

What a router can decide

Three ways to build the router, cheapest first

5.23 Agentic RAG

The step where retrieval stops being a preprocessing step and becomes something the model does, repeatedly, on purpose, until it has what it needs.

In every architecture so far, retrieval happened to the model. Some code searched, pasted the results into a prompt, and the model wrote whatever it could from that. If the search was poor, the model had no recourse: it could not look again, ask a different question, or check a second source. It could only produce the best answer available from a fixed pile of text.

Agentic RAG inverts the control flow. The model is given retrieval as a tool it can call, and a loop in which to call it. It decides what to search for, inspects what came back, and decides whether to search again. Retrieval becomes an action rather than a stage.

A worked trace, step by step

The clearest way to understand the loop is to watch one run. Take a question a single-shot pipeline cannot answer at all:

Is the laptop I ordered last month still under warranty, and does it cover the cracked screen?

This needs three facts from three places, and — crucially — the second search cannot be written until the first one returns. No amount of query rewriting fixes that, because the dependency is on data, not on phrasing.

1. Think — decompose the question "I need three things: which laptop was ordered, what warranty applies to that model, and whether accidental damage is covered. I cannot look up the warranty until I know the model." The plan itself is the first output, and it is what makes the ordering explicit.
2. Act — call the orders tool get_orders(customer_id, window="60d") returns Acer Swift 3, purchased 2025-09-14. Note this is a structured lookup, not a vector search. An agent's tools do not all have to be retrievers.
3. Observe — and use the result to write the next query Now the model knows the model name, so it can form a query that was impossible a moment ago: search_policies("Acer Swift 3 warranty period"). This is the dependency that defeats one-shot retrieval.
4. Act again — and get a partial answer The policy index returns "Acer laptops carry a 12-month manufacturer warranty." Twelve months from September is still valid, so the first half of the question is settled. The second half is not: nothing so far mentions cracked screens.
5. Judge — notice what is missing "I have the warranty period but nothing about physical damage. That is a separate policy." This self-assessment is the step that distinguishes an agent from a pipeline. A pipeline would answer now, from what it happened to have.
6. Act a third time — targeted at the gap search_policies("accidental damage cracked screen coverage") returns "Physical damage is excluded from the manufacturer warranty; see the optional protection plan."
7. Synthesise — one answer from three retrievals "Your Acer Swift 3 is still under its 12-month warranty until September 2026. However, that warranty excludes physical damage, so a cracked screen is not covered unless you purchased the protection plan." Every clause traces to a specific retrieval, so the citation machinery from 5.19 still applies.

Single-agent, multi-agent, and what each is for

The guardrails that make loops safe

A loop controlled by a probabilistic model will, given enough traffic, fail to terminate. These bounds are not optional extras; they are the difference between a demo and a production system.

5.24 Corrective RAG

Grade the retrieved documents before using them, and do something different when they are bad. A small idea with an unusually good return.

Corrective RAG, introduced by Yan and colleagues in 2024, targets one specific failure: retrieval returns something, the something is wrong or irrelevant, and the generator uses it anyway. Naive RAG has no concept of a bad retrieval — the top 5 results are simply the results, however poor.

The fix is to insert an explicit grader between retrieval and generation, and to give the system three possible verdicts rather than one path.

5.25 Self-RAG

Train the model to decide for itself when to retrieve, and to criticise its own output as it writes. Corrective RAG bolts a judge on the outside; Self-RAG moves the judgement inside.

Self-RAG, from Asai and colleagues in 2023, makes a more radical move than anything so far. Rather than surrounding the model with graders and routers, it fine-tunes the model to emit special reflection tokens alongside ordinary text. Those tokens are decisions, and the surrounding system reads them and acts.

5.26 GraphRAG

Some questions cannot be answered by any single chunk, because the answer does not exist in any single chunk. It has to be assembled from relationships spread across the whole corpus.

Every retrieval method so far shares an assumption: somewhere in your corpus there is a passage that answers the question, and the job is to find it. That assumption holds for what is the returns window for televisions. It collapses completely for which of our suppliers are affected by the port closure in Rotterdam.

No document contains that answer. The information exists as a chain: shipping records name the port, container manifests link to shipments, shipments link to purchase orders, purchase orders name suppliers. Each hop lives in a different document. Vector search retrieves the five chunks most similar to the question, which will be five documents mentioning Rotterdam — and none of them lists a supplier.

The two questions vector search cannot answer

How the graph gets built

You do not usually have a knowledge graph lying around. It is constructed from the same documents you were already chunking, using an LLM as the extractor. This is an ingestion-time cost, paid once per document rather than per query.

1. Extract entities and relationships Pass each chunk to a model with a schema: pull out the people, products, suppliers, locations and the relationships between them. The output is a list of triples such as (PO-2219, placed_with, Delta Textiles).
2. Resolve duplicates into single nodes Delta Textiles, Delta Textiles Ltd and DELTA TEXTILES LTD. must become one node or the graph fragments and traversal fails. This entity-resolution step is the hardest part of the build and the one most often underestimated.
3. Detect communities Run a clustering algorithm — Leiden is the usual choice — to find densely connected groups of nodes. These clusters correspond to themes: a supplier and all its orders, or a product family and its complaints.
4. Summarise each community Have the model write a summary of every cluster, and summaries of clusters of clusters, forming a hierarchy. This is what makes global questions answerable: the corpus-wide question is answered from a few dozen community summaries rather than from thousands of chunks.

5.27 Multi-vector retrieval and ColBERT

One vector per chunk throws away almost everything. Multi-vector methods keep a vector per token, and buy back precision that single-vector search structurally cannot reach.

Recall what a bi-encoder does: it compresses an entire 400-word passage into a single point in space, perhaps 768 numbers. That compression is lossy in a specific and damaging way. A passage covering three topics gets averaged into a vector that sits between all three and represents none of them well. Match a query about topic two against that average and the score is mediocre, even though the passage answers the question perfectly.

5.28 Contextual retrieval

A chunk torn out of its document loses the context that made it meaningful. Fixing that with one cheap trick produced one of the largest measured retrieval gains of recent years.

Consider a chunk lifted from the middle of a policy document:

This period is extended to 60 days for members, and does not apply to clearance items.

Which period? Extended from what? This chunk is nearly useless in isolation, and worse, it is unretrievable: it never says "returns", so a query about return windows will not match it. The information was in the section heading three pages up, which the chunker discarded. Section 5.6 introduced overlap as a partial remedy; overlap does not reach three pages.

The same chunk, contextualised, becomes something a retriever can actually find:

This section is from the 2025 Returns Policy, describing the standard 30-day return window for general merchandise. This period is extended to 60 days for members, and does not apply to clearance items.

5.29 RAPTOR and hierarchical retrieval

Chunks answer detail questions. Summaries answer overview questions. RAPTOR builds both from the same corpus and lets retrieval choose the level it needs.

There is a mismatch between how we chunk and how people ask. Chunking optimises for specific facts: small passages, tight topics, precise matches. But a good share of real questions are not specific at all. What are the main changes in this year's policy? has an answer that spans forty pages, and no 400-token chunk contains it.

Why indexing every level is the clever part

It would be easy to assume you must decide in advance whether a query is specific or broad, then search the right level. RAPTOR avoids that decision entirely. Because summaries and leaves live in the same index, ordinary similarity search picks the level for you.

• what is the TV return window matches a leaf chunk, because leaves contain the specific numbers and the summaries do not.
• how has the returns policy changed matches a mid-level summary, because only summaries discuss change across sections.
• what is this document about matches near the root.

5.30 Multimodal and conversational RAG

Two extensions you will meet in any real deployment: documents that are not only text, and users who ask a second question.

Multimodal retrieval

A product catalogue is photographs. A manual is diagrams. An invoice is a scanned table. Text-only pipelines silently drop all of it — and "silently" is the operative word, because ingestion reports success while the substance of the document never reaches the index.

Conversational retrieval

The second question is where naive systems fall apart. Is it waterproof? is unanswerable in isolation: what is? Embedding that string retrieves documents about waterproofing in general, and the top result may be a tent.

5.31 Choosing an architecture

Ten techniques have been laid out. You should use two or three of them. This section is about picking which.

The temptation after a survey like this is to build the most sophisticated thing described. Resist it. Every component added is another failure mode, another latency budget, another thing to evaluate and keep working. The good news is that the ordering is fairly stable across projects, because the cheap wins really are the big ones.

5.32 Retrieval metrics

You cannot improve what you do not measure, and in RAG the thing most worth measuring is the stage nobody looks at. If retrieval is broken, nothing downstream can save you.

The instinct is to evaluate the final answer, because that is what users see. The problem is that a bad answer tells you almost nothing about why. Retrieval metrics isolate the first stage so you can tell "we never found the document" apart from "we found it and wrote a poor answer" — two problems with nothing in common.

"""Retrieval metrics, implemented rather than described.

These are easy to get subtly wrong, and a wrong metric is worse than none --
it sends you optimising in the wrong direction with full confidence. Each
function here is small enough to check by eye against the worked example.
"""
from __future__ import annotations


def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Fraction of the relevant documents that made it into the top k.

    The ceiling on everything downstream: a document missing here cannot be
    reranked, cited or summarised later.
    """
    if not relevant:
        return 1.0            # nothing to find, so nothing was missed
    found = len(set(retrieved[:k]) & relevant)
    return found / len(relevant)


def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Fraction of the top k that are actually relevant."""
    if k == 0:
        return 0.0
    return len(set(retrieved[:k]) & relevant) / k


def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    """One over the position of the first relevant hit; 0 if there is none."""
    for position, doc in enumerate(retrieved, start=1):
        if doc in relevant:
            return 1.0 / position
    return 0.0


def dcg(gains: list[float]) -> float:
    """Discounted cumulative gain, with a log2 positional discount.

    Position 1 is undiscounted, position 2 is divided by log2(3), and so on,
    so a correct answer buried at rank 9 contributes far less than one at
    rank 1.
    """
    from math import log2
    return sum(gain / log2(position + 1)
               for position, gain in enumerate(gains, start=1))


def ndcg_at_k(retrieved: list[str], graded: dict[str, float], k: int) -> float:
    """NDCG for graded relevance, normalised by the best possible ordering."""
    actual = dcg([graded.get(doc, 0.0) for doc in retrieved[:k]])
    ideal = dcg(sorted(graded.values(), reverse=True)[:k])
    return actual / ideal if ideal else 0.0


# --- worked example ---------------------------------------------------------
# The ranking a retriever returned, best-first.
RETRIEVED = ["d3", "d7", "d1", "d9", "d4", "d2"]
# Binary judgements: these three documents genuinely answer the question.
RELEVANT = {"d1", "d2", "d5"}
# Graded judgements: 3 = perfect, 2 = useful, 1 = tangential, absent = useless.
GRADED = {"d1": 3.0, "d2": 2.0, "d5": 3.0, "d3": 1.0}

print(f"recall@3     {recall_at_k(RETRIEVED, RELEVANT, 3):.2f}")
print(f"recall@6     {recall_at_k(RETRIEVED, RELEVANT, 6):.2f}")
print(f"precision@3  {precision_at_k(RETRIEVED, RELEVANT, 3):.2f}")
print(f"MRR          {reciprocal_rank(RETRIEVED, RELEVANT):.2f}")
print(f"NDCG@6       {ndcg_at_k(RETRIEVED, GRADED, 6):.2f}")

recall@3     0.33
recall@6     0.67
precision@3  0.33
MRR          0.33
NDCG@6       0.51

Read those five numbers together and they diagnose the system in a way no single one could. Recall climbs from 0.33 to 0.67 as k widens, so more results genuinely help — but it stops at 0.67 because d5 was never retrieved at any depth. That is a retriever problem, and no reranker can touch it. MRR of 0.33 says the first correct hit sat at position three, so the material is being found and ranked poorly — which is a reranker problem. One metric says fix retrieval, the other says fix ranking, and both are true of different parts of this system.

5.33 Generation metrics

Retrieval metrics need labelled judgements. Generation metrics mostly need another model, which is convenient, cheap, and quietly dangerous.

Once the right documents are in the prompt, three things can still go wrong: the answer can contradict them, ignore the question, or rest on the wrong subset of them. Those map onto three measurements, together commonly called the RAG triad.

5.34 Golden sets and regression testing

Metrics need something to measure against. The golden set is that something, and building a good one is the least glamorous, highest-value work in the whole chapter.

Building one that is actually useful

1. Start from real queries, not imagined ones Pull questions from support tickets, search logs and chat transcripts. Questions invented by the team are systematically too clean: correctly spelled, well formed, and using internal vocabulary that no customer says. Real queries contain typos, slang and partial product names, which is exactly where systems break.
2. Cover the categories deliberately Aim for a spread: simple lookups, multi-hop questions, comparisons, questions with no answer in the corpus, ambiguous questions, and adversarial ones. The unanswerable cases are essential — they are the only way to measure the abstention behaviour from 5.20.
3. Label the relevant documents, not just the answer For each question record which chunks should be retrieved. This is the tedious part and the part that makes retrieval metrics possible. Without it you can only evaluate end-to-end, which tells you something is wrong but never what.
4. Use a model to draft, a human to approve Have an LLM propose questions from your documents and suggest relevance labels, then review them. Drafting is the slow part; correcting is fast. Never ship labels no human has read, or you are measuring your model against its own opinions.
5. Keep it versioned and growing Every production failure becomes a new test case. That single habit turns your golden set into an asset that compounds: the system becomes permanently incapable of regressing on any bug you have already fixed.

5.35 Caching, latency and cost

A RAG system that works beautifully in a notebook can be unaffordable and too slow in production. Both problems have the same small set of answers.

Where the time actually goes

Three kinds of cache, in increasing order of subtlety

1. Exact-match response cache Hash the normalised query; if you have answered it before and the corpus has not changed, return the stored answer. Trivial to build, and on a storefront where thousands of people ask about the returns policy every day it removes an enormous amount of traffic for free.
2. Semantic cache Embed the query and look for a previous question within a tight similarity threshold. Catches paraphrases that exact matching misses. Set the threshold conservatively: can I return a TV and can I return a TV stand are close in embedding space and have different answers. A semantic cache that is too loose serves confidently wrong answers, which is worse than no cache.
3. Prompt cache Provider-side caching of a shared prompt prefix. Your system instructions and few-shot examples are identical on every request, so the provider keeps them warm and charges a fraction for the repeated part. This is what makes contextual retrieval (5.28) affordable, and it is the reason the system prompt belongs at the very top of the context, exactly where 5.17 put it.

5.36 Security and prompt injection

RAG systems have a security property most applications do not: they deliberately take untrusted text and place it into a privileged instruction channel.

This is worth stating plainly. Your pipeline fetches documents and pastes them into the same context that holds your system instructions. The model has no reliable way to tell the two apart — both are simply tokens. If an attacker controls any text that can be retrieved, they control part of your prompt.

The three risks, and what actually helps

5.37 Observability and failure triage

When a RAG answer is wrong, the answer itself rarely tells you why. Everything you need is in the intermediate steps, and only if you logged them.

A traditional web service fails loudly: an exception, a stack trace, a 500. A RAG pipeline fails fluently. It returns a well-formed, confident, plausible paragraph that happens to be wrong, and every component reports success. Nothing in your error rate will move. This is why observability here is not optional instrumentation but the primary debugging tool.

The three-bucket triage

Sample fifty failing queries and sort each into exactly one bucket. This is the routine referred to throughout the chapter, and it is what turns a vague "quality is poor" into a specific engineering task.

5.38 Key takeaways

5.39 Interview drills

RAG is the topic most likely to be probed in depth, because almost everyone claims it and few can explain the failure modes. Strong answers here share one quality: they name a specific mechanism and the trade-off it carries, rather than listing techniques.

Expand all

Where this leaves you

You can now build a retrieval pipeline that finds the right evidence, arranges it so the model can actually use it, cites it in a way that survives being checked, and refuses when the evidence is not there. More importantly, you can measure each of those stages separately, which is what makes the difference between improving a system and changing it.

The thread running through this chapter is that RAG is not one technique but a sequence of decisions, each with a failure mode attached. Chunking decides what can be found. Retrieval decides the ceiling. Reranking decides what the model sees. Ordering decides what it notices. Grounding and citation decide whether anyone can trust the result. Skip the measurement and you are tuning blind; skip the diagnosis and you are rebuilding the wrong stage.

Chapter 6 moves from retrieval to the systems that act on it: agents, tool use, and the orchestration frameworks that hold multi-step workflows together — including what happens when a reasoning loop meets a real production environment with budgets, latency ceilings and users who will not wait.