Chapter 17 — Statistics for AI Engineers

[!] Why this chapter exists, and why it is here rather than at the start

Every chapter so far has quietly depended on statistics. Chapter 5 evaluated a retrieval pipeline. Chapter 6 reasoned about p99 latency. Chapter 8 computed compounding per-step reliability. Chapter 16 estimated load. Each of those used a statistical idea without stopping to examine it, on the grounds that the surrounding engineering was the point.

This chapter stops to examine them. It is placed late deliberately: the concepts land far better attached to systems you have already built than as abstract preliminaries. You now have a latency distribution, an evaluation set, and a retrieval metric to think about — which is exactly what the sections below are for.

[i] The stance

Most statistics teaching optimises for computing the right number. This chapter optimises for not being fooled, because that is the failure mode that actually occurs in engineering work. The recurring question is not "what is the p-value" but "what would have to be true for this conclusion to be wrong, and did I check?"

Every figure in this chapter was computed rather than recalled, with the code shown where it is short enough to be useful. Where a claim is commonly repeated and wrong, it is simulated and the simulation reported.

17.1 Why an AI engineer needs this

You do not need to derive anything. You need to avoid a specific, short list of mistakes that cause engineers to ship the wrong model, believe a result that isn't there, and report a number that misleads their own team.

Where this has already come up in the course
You did this	It was really
Read a p99 latency target (6.4, 16.21)	A percentile, chosen because the mean hides the tail entirely
Measured recall@10 on 200 queries (5.31)	A sample estimate with an uncertainty nobody reported
Compared two rerankers and picked one (5.29)	A hypothesis test, run informally and probably underpowered
Computed compounding step reliability (8.16)	Independent probabilities multiplying — and the independence assumption mattered
Estimated peak load as 5× average (16.18)	A distributional claim about traffic, worth checking against real data

[+] The four mistakes this chapter is designed to prevent

Reporting a mean for something that isn't symmetric Latency, order value, session length and token counts are all skewed. The mean of a skewed distribution describes almost nobody.
Treating a difference as real without asking how big the sample was Two rerankers scoring 0.71 and 0.73 on 200 queries have not been distinguished.
Reading a p-value as the probability the result is real It is not that, and the difference changes what decision you should make.
Forgetting the base rate A 95%-accurate detector for a 0.1% event is mostly wrong when it fires, and this is the single most common misunderstanding in applied ML.

17.2 The mean is often the wrong number

The mean is the default summary in every dashboard, and for the data an AI system actually produces it is frequently the least informative choice available.

[def] Mean, median, mode

The mean is the total divided by the count. The median is the middle value when sorted — half above, half below. The mode is the most common value. On a symmetric distribution all three roughly coincide, which is why they are easy to confuse. On a skewed one they separate, and the gap between them is the information.

[retail] Ten thousand real search latencies

the same data, six summariestext

mean       103.1 ms      <- what the dashboard shows
median      80.8 ms      <- what a typical user experiences

p50         80.8 ms
p90        131.0 ms
p95        153.9 ms
p99        872.2 ms      <- what the angry users experience
p99.9     1954.1 ms

only 25.4% of requests are slower than the mean

The mean sits at 103 ms, which describes no actual experience: it is dragged upward by a 2% minority of slow requests, and three quarters of users are faster than it. Report the mean and you overstate typical latency by 28% while completely concealing that one request in a hundred takes nearly a second.

[!] The mean is not robust, and that is the whole problem

One extreme value moves the mean and barely moves the median. A single 60-second timeout in a batch of a thousand 80 ms requests lifts the mean by 60 ms and the median by nothing. That sensitivity is occasionally what you want — the mean is the right choice when you care about a total, like monthly token spend, because there the outlier genuinely is part of the bill. It is the wrong choice whenever you are trying to describe a typical case.

[+] A rule that survives contact with production

Use the mean for things you sum, the median for things you experience. Total cost, total tokens, total revenue: mean. Latency, session length, order value, time to resolution: median plus percentiles. And never report a single number for a distribution you have not looked at — the shape decides which summary is honest, which is why 17.4 spends its time on shapes.

17.3 Spread, and why it matters more

Two systems with identical mean latency can offer completely different experiences. The average tells you where the distribution sits; the spread tells you how much you can trust that summary.

Variance: The average squared distance from the mean. Squaring keeps everything positive and makes the mathematics work cleanly, at the cost of being in unintelligible units — variance of latency is measured in milliseconds squared, which means nothing to anyone.
Standard deviation: The square root of the variance, so it is back in the original units. This is the one to quote. Roughly: the typical distance of a value from the mean.
Interquartile range (IQR): The distance from the 25th to the 75th percentile — the width of the middle half. The robust alternative to standard deviation, and the right choice for the skewed data in 16.2.

[!] The 68–95–99.7 rule only applies to normal data

You will hear that 95% of values fall within two standard deviations of the mean. That is a property of the normal distribution specifically, not a law of nature. For the latency data above, mean plus two standard deviations lands nowhere near p95, because the distribution has a long right tail and no left one. Applying the rule to skewed data produces confident, wrong thresholds — and this is a common source of badly set alerts that either never fire or fire constantly.

[+] Coefficient of variation, for comparing unlike things

A standard deviation of 50 is large for a metric averaging 80 and trivial for one averaging 50,000. The coefficient of variation — standard deviation divided by the mean — is dimensionless, so it lets you say "embedding latency is more variable than search latency" without the units getting in the way. Useful when deciding which component's inconsistency is worth chasing first.

17.4 Distributions you will actually meet

Five shapes cover almost everything an AI system generates. Recognising which one you are looking at determines which summary is honest, which test is valid, and whether your intuition applies at all.

The five, and where each shows up in this course
Distribution	Shape	You have already seen it as
Normal	Symmetric bell. Mean = median.	Rarer in raw data than you'd think. Mostly appears as the distribution of averages (17.7), which is where it earns its importance.
Log-normal	Right-skewed; normal after taking logs.	Latency (17.2), session length, order value. The default shape for "duration of something."
Power law	Extreme right skew; a few values dominate the total.	Query frequency, product popularity, tenant size (16.22). A handful of items account for most volume.
Bernoulli / binomial	Counts of yes/no over n trials.	Conversion, click-through, whether a retrieval was relevant (5.31), whether a step succeeded (8.16).
Poisson	Counts of events in a fixed window.	Requests per second, errors per hour. The basis for reasoning about queue arrivals (16.15).

[!] Power laws break averages entirely

For a sufficiently heavy-tailed power law, the sample mean does not converge usefully: take more data and it keeps climbing, because you keep encountering larger extremes. This is not a theoretical curiosity. "Average queries per tenant" on a platform where one tenant sends 40% of traffic is a number that describes nobody and changes every month. For power-law data, report the median, the top-decile share, and the maximum — and size infrastructure for the largest tenant, not the average one.

[+] Log-normal has a practical consequence worth knowing

If a metric is log-normal, then its logarithm is normal — so taking logs first makes the standard toolkit valid again. Averaging log-latency and exponentiating gives the geometric mean, which is far more stable than the arithmetic mean for skewed data. This is also why latency charts are so often drawn on a log axis: it turns a hopeless smear against the bottom of the plot into something you can actually read.

17.5 Percentiles and the latency tail

Percentiles are how the industry talks about latency, and they carry two traps that catch experienced engineers.

[def] Percentile

The p99 is the value below which 99% of observations fall. If p99 latency is 872 ms, then 99 requests in 100 completed faster than that, and one did not. It says nothing about how much slower that one was — which is why p99.9 and the maximum are separate, useful numbers.

[!] Trap one: percentiles do not average

This is the single most common statistical error in production monitoring. You cannot average p99 values across servers, or across time buckets, to get an overall p99. The average of the p99s of ten servers is not the p99 of the combined traffic, and it is usually a substantial underestimate.

The reason is that a percentile is a property of a distribution, not a quantity that adds. To combine, you need the underlying data or a mergeable sketch — which is why monitoring systems store histograms or t-digests rather than pre-computed percentiles. If your dashboard shows an "average p99", it is showing a number that corresponds to nothing.

[!] Trap two: the user's experience is worse than your p99

A single page load that makes twenty backend calls will exceed your p99 latency on at least one of them about 18% of the time — because 1 - 0.99²⁰ is 0.182. The page is as slow as its slowest call, so a 1-in-100 backend event becomes a 1-in-5 user event. This is exactly the compounding from 8.16 wearing different clothes, and it is why tail latency gets disproportionate attention in systems with fan-out (16.5).

[retail] Which percentile to actually target

p50 tells you whether the system is fundamentally fast enough. p95 to p99 is where SLOs belong, because that is where users notice and complain. p99.9 is where you find the genuinely broken paths — GC pauses, cold caches, a retry that fired. Chasing p99.9 on a search box is usually poor prioritisation; chasing it on a payment API is not. Pick the percentile from the consequence of being slow, not from convention.

17.6 Sampling, and how it goes wrong

Every metric you have ever reported was computed on a sample. The question is never whether you sampled, but whether the sample resembles the thing you want to conclude about — and that is a question about how it was collected, which no amount of statistics can fix afterwards.

[!] The four sampling failures that show up in AI work

Selection bias. Your evaluation set is 200 queries someone wrote by hand. They are cleaner, better spelled and more grammatical than real queries, so every number measured on them is optimistic (5.31).
Survivorship bias. You analyse latency of completed requests. The ones that timed out are absent — precisely the slowest ones, removed from the measurement of slowness.
Period bias. You benchmark on Tuesday afternoon traffic and deploy for Black Friday. The distribution of queries, not just their volume, is different.
Feedback loops. You train on clicks from a ranking your own model produced. The model is now learning from its own past behaviour, and will confidently reinforce whatever it already did (18.19).

[+] The only reliable defence is boring

Sample randomly from real production traffic over a period long enough to include the weekly cycle, and stratify if you need guaranteed coverage of a rare segment. A stratified sample draws a fixed number from each group — so a market representing 2% of traffic still gets enough queries to say something about, and you weight the groups back to their true proportions when combining. This is how you avoid discovering after launch that the model was never evaluated on Mexico.

17.7 The central limit theorem, usefully

[def] The one theorem worth knowing

The central limit theorem says that the distribution of the sample mean approaches a normal distribution as the sample grows, almost regardless of the shape of the underlying data. Note carefully what it does not say: your data does not become normal, and never will. Latency stays log-normal forever. It is the average of many latencies that behaves normally.

[+] Why this is the load-bearing idea in the chapter

It is what licenses everything in sections 17.8 to 16.16. Because sample means are approximately normal, you can attach a standard error to one, build a confidence interval around it, and run a t-test comparing two of them — all without knowing or caring what shape the raw data has. Nearly every routine statistical procedure is cashing this cheque.

[!] "n > 30" is folklore, and the tail decides

The convergence depends on skew. For roughly symmetric data, 30 is plenty. For the heavy-tailed distributions AI systems produce — latency, order value, tenant size — a few hundred may not be enough, because a single extreme value still dominates the average. And for percentiles the theorem says nothing at all: the CLT applies to means, not to p99. If you need uncertainty on a percentile, use the bootstrap (17.10).

17.8 Standard error versus standard deviation

These are confused constantly, including in production dashboards and in interviews, and the distinction is genuinely simple once stated plainly.

Standard deviation (SD): How spread out the data is. A property of the population. Collecting more data does not reduce it — you just measure it more precisely. If order values vary a lot, they vary a lot.
Standard error (SE): How uncertain your estimate of the mean is. Equal to SD / √n. It shrinks as you collect more data, because a bigger sample pins down the average more tightly.

[!] The square root is the expensive part

Because SE falls with √n, halving your uncertainty requires four times the data, and a tenfold improvement requires a hundredfold sample. This single fact explains why experiments take as long as they do, why small effects are so expensive to detect (17.15), and why variance reduction techniques like CUPED (19.17) are worth real engineering effort — they buy precision without buying traffic.

17.9 Confidence intervals

A point estimate with no interval around it is an assertion, not a measurement. The interval is what tells your reader how much to trust the number.

[retail] The 200-query evaluation set, revisited

recall@10 measured on n queriestext

recall@10 = 0.72 measured on 200 queries
95% CI = [0.658, 0.782]      +/- 6.2 percentage points

precision of the estimate by sample size:
  n =   200      +/- 6.2pp
  n =  1000      +/- 2.8pp
  n =  5000      +/- 1.2pp

Chapter 5 compared rerankers scoring 0.71 and 0.73 on a set this size. Those intervals overlap almost entirely — a formal test gives p = 0.66, which is no evidence of a difference whatsoever. Detecting a gap that small at 80% power needs about 7,900 queries per arm. The honest conclusion from 200 queries is "these two rerankers are indistinguishable on this evidence," and that is a genuinely useful thing to be able to say.

[!] What a 95% confidence interval actually means

It does not mean there is a 95% probability the true value lies in this particular interval — under the frequentist framing the true value is fixed, and either it is in there or it isn't. It means: if you repeated the whole experiment many times, 95% of the intervals so constructed would contain the true value. In practice people read it the first way, and for most decisions that reading is harmless — but if you want a statement of the first kind, you want a Bayesian credible interval (17.20), which genuinely does mean that.

[+] The habit worth forming

Report every metric as an interval, in every document and every dashboard. "Recall improved from 0.71 to 0.73" invites a decision. "Recall was 0.72 ± 0.06, then 0.73 ± 0.06" invites the correct decision, which is to collect more data before concluding anything. The interval is not pedantry — it is the difference between reporting a measurement and reporting a rumour.

17.10 The bootstrap

Sometimes the quantity you care about has no tidy formula for its standard error — a p99, a median, an NDCG, the ratio of two metrics. The bootstrap handles all of them with one idea and no mathematics.

[def] Resampling, in one paragraph

Draw a new sample of the same size from your data with replacement, compute your statistic on it, and repeat a few thousand times. The spread of those values estimates the uncertainty of the original statistic. The intuition: your sample is your best available picture of the population, so resampling from it simulates the variation you would have seen by collecting fresh data.

bootstrap_p99.pypython

import numpy as np

rng = np.random.default_rng(0)
boots = [np.percentile(rng.choice(latencies, len(latencies)), 99)
         for _ in range(2000)]

print(np.percentile(boots, [2.5, 97.5]))
# p99 = 765 ms, 95% CI [330, 969]

[!] That interval is enormous, and that is the lesson

A p99 estimated from 1,000 requests has a 95% interval spanning 330 to 969 ms — roughly a factor of three. This is not a flaw in the bootstrap; it is the truth about extreme percentiles, which are computed from a handful of observations no matter how large the sample. p99 from 1,000 requests is 10 data points. Anyone reading a daily p99 chart and reacting to day-to-day movement is mostly reacting to noise, and the bootstrap is how you demonstrate that to them.

[+] When to reach for it

Use the bootstrap whenever the statistic is not a simple mean: percentiles, medians, ranking metrics like NDCG and MRR (5.31), correlation coefficients, or any ratio. It costs a few seconds of compute and removes the need to look up whether a formula exists. The main caveat is that it cannot rescue a biased sample — resampling bad data gives you a precise estimate of the wrong thing.

17.11 The logic of a hypothesis test

Hypothesis testing has a reputation for being fiddly, mostly because it is taught as a procedure rather than an argument. The argument itself is short, and once you have it the procedure follows.

[def] The argument, in four steps

Assume nothing is happening The null hypothesis is that the two things are identical: the new reranker is no better, the variant does not convert differently.
Ask how surprising your data would be under that assumption If the reranker really were no better, how often would random noise alone produce a gap this large or larger?
That "how often" is the p-value A small p-value means your observation would be unusual if nothing were happening.
Decide Below your threshold, you reject the null and act as if something is happening. Above it, you have not shown anything — which is not the same as having shown there is nothing.

[+] It is a proof by contradiction with a probability attached

That is genuinely all it is. Assume the boring explanation, show the data would be improbable under it, conclude the boring explanation is doubtful. The asymmetry that confuses people follows directly: you can accumulate evidence against the null, but never evidence for it, in the same way a failed search for a counterexample does not constitute a proof.

17.12 What a p-value is not

This section exists because the misreadings are near-universal, they appear in post-launch write-ups, and each one leads to a different bad decision.

Four misreadings, and what the p-value actually says
People think p = 0.03 means	The correction
"There is a 97% chance the effect is real."	No. The p-value is computed assuming the null is true. It cannot also tell you the probability that the null is false — that requires a prior (17.20).
"There is a 3% chance this was a fluke."	No. It is the probability of data this extreme given no real effect, not the probability of no real effect given the data. Reversing a conditional probability changes its value entirely.
"The effect is large / important."	Unrelated. With enough traffic, a 0.01% lift produces a tiny p-value. Significance is about detectability, not size (17.14).
"p = 0.06 means there is no effect."	No. It means you failed to detect one, which at small sample sizes is the expected outcome even for real effects. Absence of evidence is not evidence of absence.

[!] 0.05 is a convention, not a discovery

The threshold was a rule of thumb suggested by Fisher in the 1920s, and nothing about the world changes between p = 0.049 and p = 0.051. Treating it as a bright line is what produces the pathology of researchers — and engineers — nudging analyses until they land on the correct side of it. The threshold should follow the cost of being wrong: a change that is cheap to reverse deserves a looser one; a change to the payment flow deserves much stronger evidence than 0.05.

17.13 Choosing a test

The choice is nearly mechanical once you know what kind of data you have and how the groups relate. This table covers the overwhelming majority of real cases.

Which test, and when
Situation	Test	Typical use
Two proportions, independent groups	Two-proportion z-test proportions_ztest	Conversion rate in an A/B test (chapter 19)
Two means, independent groups	Welch's t-test ttest_ind(equal_var=False)	Revenue per session between variants
Two means, same units measured twice	Paired t-test ttest_rel	Two rerankers scored on the same query set
Skewed data, small sample, means unreliable	Mann–Whitney U mannwhitneyu	Latency comparison on a few hundred requests
Categorical counts across categories	Chi-square chi2_contingency	Whether error types differ between two model versions
Three or more groups	ANOVA, then post-hoc	Four prompt variants at once — and see 17.16 before you do this

[+] Default to Welch, not Student

The classic t-test assumes both groups have equal variance. Welch's version does not, costs essentially nothing in power when variances happen to be equal, and is substantially more reliable when they are not. In SciPy that means passing equal_var=False, and there is very little reason ever to omit it.

[retail] Pairing is worth more than a bigger sample

Chapter 5's reranker comparison scores both models on the same queries. That means the observations are paired, and using a paired test removes the enormous query-to-query variation — some queries are simply easier than others — leaving only the difference you care about. The effect is dramatic:

identical data, two teststext

200 queries, NDCG per query
  model A mean = 0.7062
  model B mean = 0.7261        difference = +0.0199

  unpaired t-test    p = 0.34         "no significant difference"
  PAIRED   t-test    p = 0.0000031    clearly significant

  Cohen's d: 0.095 unpaired  ->  0.378 paired

Same numbers, opposite conclusions. The unpaired test is drowning a real +0.02 improvement in variance that pairing eliminates entirely. If the same unit appears in both groups, pairing is not an optimisation — using the unpaired test is a mistake.

17.14 Effect size and why it outranks significance

[def] Effect size

Significance answers "can I detect it?". Effect size answers "is it big enough to matter?" — and only the second question has business consequences. The common measures are the raw difference (most interpretable), the relative lift (most quoted), and Cohen's d, the difference expressed in standard deviations, which allows comparison across metrics with different units.

[!] At scale, everything becomes significant

With 6 million searches a day (16.18), a 0.02% conversion difference will eventually produce p < 0.001. It is real, it is detectable, and it is worth nothing — a rounding error that costs a quarter of engineering time to maintain. This is why mature experimentation programmes define a minimum detectable effect in advance (19.9): the smallest lift that would actually change the decision. Anything below it is not a win, however significant.

[+] Report all three, in this order

The absolute difference, the relative lift, and the confidence interval — then the p-value if anyone still wants it. "Conversion rose 0.3 percentage points, a 10% relative lift, 95% CI [0.1pp, 0.5pp]" supports a decision. "p = 0.02" does not, because it contains no information about magnitude at all.

17.15 Errors, power, and the trade-off

Two ways to be wrong, and they trade against each other. Understanding which one your situation punishes more is what turns a threshold from a convention into a decision.

The two errors, and what each costs in practice
Error	Meaning	Cost in an AI system
Type I (false positive, α)	You conclude there is an effect when there isn't.	Ship a change that does nothing. Ongoing complexity, maintenance, and a team that believes something false.
Type II (false negative, β)	You miss a real effect.	Discard a genuine improvement. Invisible, unrecorded, and therefore chronically underweighted.

[def] Power

Power is 1 − β: the probability of detecting an effect that genuinely exists. The convention is 80%, which is worth saying out loud — a conventionally designed experiment misses one real effect in five. Power rises with sample size and with effect size, and falls as you demand a stricter α.

[!] An underpowered test is worse than no test

If power is 20%, then four times in five a real effect goes unnoticed — but worse, the results that do reach significance are systematically the ones where noise happened to inflate the estimate. So the experiment produces a small number of exaggerated wins and a large number of false reassurances, while the team believes it is measuring carefully. This is the winner's curse, and it is why effect sizes from small experiments consistently fail to replicate at scale.

[+] Compute power before running, never after

Power analysis answers "how much data do I need to detect the smallest effect I would act on?" and it belongs at the design stage, where it can still change what you do (19.9). Computing it afterwards to explain a null result — "observed power" — is circular, because it is just the p-value re-expressed. If the answer is "eleven weeks of traffic", that is information you needed before starting, not after.

17.16 Multiple comparisons

Run one test at α = 0.05 and you accept a 5% false positive rate. Run twenty and the arithmetic stops being reassuring.

[retail] The dashboard problem

probability of at least one false positivetext

metrics tested    P(>=1 false positive)    Bonferroni alpha
       1                    5.0%                0.0500
       5                   22.6%                0.0100
      10                   40.1%                0.0050
      20                   64.2%                0.0025

A results dashboard showing twenty metrics for an experiment that changed nothing will display at least one significant result about two times in three. Teams then explain that result, build a story around it, and ship. This is not a rare pathology — it is the default outcome of looking at a lot of metrics.

Corrections, and when each is appropriate
Method	What it controls	Use when
Bonferroni	Divide α by the number of tests.	Few tests, and positive is expensive. Simple, conservative, costs power.
Benjamini–Hochberg	Controls the proportion of discoveries that are false, rather than the chance of any.	Many tests where some false positives are tolerable — exploratory analysis, feature screening.
Pre-registration	Nothing statistically; it removes the problem instead.	The best answer. Declare one primary metric before running, and everything else is explicitly exploratory.

[!] Peeking is the same problem, hidden in time

Checking an experiment repeatedly and stopping when it looks significant is multiple comparisons wearing a disguise — each check is another chance to cross the threshold. Simulated under a true null, where the correct false positive rate is 5%:

2,000 simulated A/A teststext

times checked     false positive rate
      1                   4.3%
      5                  14.7%
     10                  19.1%
     20                  25.2%

Checking daily for three weeks turns a 5% error rate into roughly 25%. The experiment is comparing a variant against itself and still declares a winner a quarter of the time. Fixed sample sizes, or sequential tests designed for continuous monitoring, are the two honest options (19.13).

17.17 Correlation, and its limits

Correlation measures how much two things move together. It is useful, widely reported, and quietly limited in ways that matter for feature engineering and for reading dashboards.

Pearson correlation: Measures linear association, from −1 to +1. The default, and the one that silently fails on curved relationships.
Spearman correlation: Pearson applied to ranks rather than values. Captures any monotonic relationship and is robust to outliers — usually the better default for the skewed data in 16.4.
R-squared: The square of the correlation: the fraction of variance in one variable explained by the other. A correlation of 0.5 explains 25% of the variance, which is a good deal less impressive than 0.5 sounds.

[!] A correlation of zero does not mean unrelated

Pearson only detects straight lines. A perfect U-shaped relationship — latency rising at both very small and very large batch sizes, say — can produce a correlation of almost exactly zero while being entirely deterministic. Anscombe's quartet is the classic demonstration: four datasets with identical means, variances and correlations that look nothing alike when plotted. Plot the data before trusting the coefficient.

[+] Where this bites in ML work

Feature selection by correlation with the target discards features that are useless alone but valuable in combination, and keeps features that are individually predictive but redundant with each other. It is a reasonable first filter and a poor final one — which is why 18.11 prefers model-based importance, and why the leakage check in 18.9 matters more than either.

17.18 Confounding and Simpson's paradox

[def] Confounder

A confounder is a third variable that influences both things you are comparing, creating an association between them that is not causal. Device type, customer tenure, time of day and market are the usual suspects in retail data, and they are usually correlated with each other too.

[retail] Simpson's paradox, with real arithmetic

A ranking change, B, is tested against control A. B wins on mobile. B wins on desktop. B loses overall:

conversion by segmenttext

            variant A              variant B          winner
mobile       40 / 2000 = 2.00%    207 / 9000 = 2.30%     B
desktop     480 / 8000 = 6.00%     65 / 1000 = 6.50%     B

POOLED      520 /10000 = 5.20%    272 /10000 = 2.72%     A

No arithmetic error: B genuinely converts better in both segments and worse overall. The cause is the traffic mix — A's users are 80% desktop, B's are 90% mobile, and desktop converts three times better regardless of variant. The pooled number is measuring the device split, not the ranking change. If this were a real experiment, the randomisation was broken, and no analysis can rescue it.

[!] The uncomfortable part

There is no purely statistical rule for choosing between the segmented and pooled answers — the data is identical and both are arithmetically correct. Deciding requires knowing why the groups differ, which is domain knowledge, not computation. Here the split is an artefact of broken assignment, so the segmented view is right. If instead the variant itself had caused the device shift, pooling would be right, because that shift is part of the effect. Same numbers, opposite conclusions, decided entirely outside the statistics.

17.19 Causation, and what earns the word

"Correlation is not causation" is repeated so often it has stopped carrying information. The useful version is knowing what evidence does support a causal claim.

Evidence for causation, weakest to strongest
Design	Strength
Observational correlation	Weakest. Any number of confounders, and the direction of causation is unknown.
Controlling for known confounders	Better, and limited by imagination; you can only adjust for confounders you thought of and measured.
Natural experiment	Something external assigns the treatment quasi-randomly. Good when available, rarely available.
Randomised controlled experiment	Strongest. Randomisation balances all confounders, including the ones nobody thought of. This is why A/B testing exists (chapter 19).

[+] Why randomisation is so powerful

It does not require you to know what the confounders are. Assign users randomly and device type, tenure, market and every unmeasured factor are balanced in expectation. That single property is what makes a modest experiment more convincing than an elaborate observational analysis on a hundred times the data — and it is why the Simpson's paradox above is a symptom of broken randomisation rather than an argument against experiments.

17.20 Bayes and the base rate problem

This is the most consequential idea in the chapter for anyone shipping a classifier, and it is the one most reliably got wrong — including by people who know the formula.

[def] Bayes' theorem, stated for engineers

The probability of a cause given evidence depends on three things: how likely the evidence is under that cause, how likely it is under every other cause, and how common the cause was to begin with. That last term is the base rate, and it is the one people drop.

[retail] A very good guardrail classifier that is mostly wrong

A prompt-injection detector (15.13) with 95% recall and a 1% false positive rate, run over a million requests where 0.1% are genuine attacks:

one million requeststext

genuine attacks     1,000  x 95% recall  ->    950 caught
benign requests   999,000  x  1% FPR     ->  9,990 false alarms

alerts raised = 950 + 9,990 = 10,940
precision     = 950 / 10,940 = 8.7%

Fewer than one alert in eleven is a real attack. Nothing is wrong with the classifier — 95% recall at 1% false positives is a genuinely good model. The problem is that 1% of an enormous benign population dwarfs 95% of a tiny malicious one. Any team asked to review these alerts will begin ignoring them within a week, which is how a working detector produces no security benefit at all.

[+] What to do about it

Report precision at your actual base rate, never accuracy, and never precision from a balanced test set that does not resemble production (18.13).
Drive the false positive rate down, not recall up. At a 0.1% base rate, FPR is the term that dominates precision. Going from 1% to 0.1% FPR takes precision from 8.7% to about 49%.
Stack weak signals. Two independent detectors both firing is far more informative than one, which is the real argument for the layered checkpoints in 14.11.
Route by confidence. High-confidence alerts block; low-confidence ones log for review. A single threshold throws away the model's own uncertainty.

17.21 Where statistics meets the AI stack

The chapter, mapped back onto systems you have built
Concept	Where it applies
Percentiles, not means (17.5)	Every latency SLO, and the reason p99 gets its own dashboard (6.4, 16.21)
Confidence intervals (17.9)	Every retrieval metric in chapter 5; the reason 200 queries cannot separate two rerankers
Bootstrap (17.10)	Uncertainty on NDCG, MRR and p99 — none of which have a convenient formula
Paired tests (17.13)	Any model comparison run on a shared evaluation set
Multiple comparisons (17.16)	Evaluating a prompt change across twelve metrics at once
Base rates (17.20)	Every guardrail classifier, hallucination detector and anomaly alert (15.13)
Randomisation (17.19)	The entire justification for chapter 19

[+] The habit this chapter is really teaching

Before believing any number, ask three questions: how was this sampled, how uncertain is it, and what else could produce this pattern. Those three cover selection bias, sample size, and confounding — which between them account for nearly every wrong conclusion in applied work. The formulas are lookups; the questions are the skill.

17.22 Key takeaways

Use the mean for things you sum, the median for things you experience. Total spend: mean. Latency, order value, session length: median and percentiles.
The mean of a skewed distribution describes nobody. In the worked example only 25% of requests were slower than the mean, while p99 was eight times the median.
The 68–95–99.7 rule is a property of the normal distribution, not of data. Applying it to latency produces confidently wrong alert thresholds.
Percentiles do not average. The mean of ten servers' p99 values is not the p99 of the combined traffic, and an "average p99" on a dashboard is a meaningless number.
Fan-out multiplies the tail. A page making twenty backend calls exceeds p99 on at least one of them 18% of the time.
Power laws break averages. Where one tenant sends 40% of traffic, size for the largest, not the average.
The CLT is about the distribution of the sample mean, not your data. Your latency stays log-normal forever, and the theorem says nothing about percentiles.
Standard deviation describes the data; standard error describes your uncertainty about its mean. Only the second shrinks with more data.
Halving uncertainty costs four times the data. This one fact explains experiment durations and why variance reduction is worth engineering effort.
Report intervals, not point estimates. Recall of 0.72 on 200 queries is 0.72 ± 0.062, which is a different claim.
Two rerankers at 0.71 and 0.73 on 200 queries are indistinguishable — p = 0.66. Separating them properly needs roughly 7,900 queries per arm.
Bootstrap anything without a formula: percentiles, medians, NDCG, ratios. A p99 from 1,000 requests is really 10 data points, and its interval spans a factor of three.
A p-value is the probability of the data given no effect, not the probability of no effect given the data. Reversing that conditional changes the answer.
Significance is detectability, not importance. At 6M searches a day a worthless 0.02% difference reaches p < 0.001.
If the same unit appears in both groups, pair the test. In the worked example pairing moved the same data from p = 0.34 to p = 0.000003.
Default to Welch's t-test. It costs almost no power when variances are equal and is far safer when they are not.
80% power means missing one real effect in five, and an underpowered experiment produces exaggerated wins rather than merely fewer of them.
Twenty metrics means a 64% chance of a false positive on an experiment that changed nothing. Pre-register one primary metric.
Peeking is multiple comparisons hidden in time. Checking twenty times takes the error rate from 5% to about 25%.
Zero correlation does not mean unrelated; Pearson only sees straight lines. Plot it.
Simpson's paradox has no statistical resolution. Choosing between pooled and segmented requires knowing why the groups differ.
Randomisation balances confounders you never thought of. That is why a modest experiment beats an elaborate observational study.
Base rates dominate rare-event detection. A 95%-recall, 1%-FPR detector on a 0.1% event is right 8.7% of the time it fires.
For rare events, cut false positives rather than chasing recall. Moving FPR from 1% to 0.1% takes precision from 8.7% to 49%.
Ask three questions of any number: how was it sampled, how uncertain is it, and what else could produce this pattern.

[i] Vocabulary check

You should be able to explain: mean versus median, variance, standard deviation, IQR, coefficient of variation, log-normal, power law, Bernoulli, Poisson, percentile, selection bias, survivorship bias, stratified sampling, central limit theorem, standard error, confidence interval, bootstrap, null hypothesis, p-value, Type I and Type II error, power, minimum detectable effect, Cohen's d, Welch's t-test, paired t-test, Mann–Whitney U, Bonferroni, Benjamini–Hochberg, peeking, Pearson versus Spearman, R-squared, confounder, Simpson's paradox, and base rate.

17.23 Interview drills

Statistics questions in an engineering interview are testing whether you can be trusted with a metric, not whether you remember formulas. These answers are written the way a strong candidate talks: the intuition first, then the caveat nobody asked for.

1. Our mean latency is 100 ms. Is that good?

I can't tell from that number, and I'd push back on it being the headline. Latency is right-skewed, so the mean is pulled upward by a small number of slow requests and describes almost nobody's actual experience.

I'd want the median and the tail. In a realistic distribution I've worked with, the mean was 103 ms while the median was 81 and p99 was 872 — so the mean overstated the typical experience by nearly 30% and completely hid that one request in a hundred took most of a second. Only about a quarter of requests were slower than the mean. The number I'd actually put an SLO on is p95 or p99, chosen based on what being slow costs us.

2. Can I average p99 across my ten servers?

No, and this is one of the most common monitoring errors. A percentile is a property of a distribution, not a quantity that adds, so the average of ten p99s is not the p99 of the combined traffic — it's usually an underestimate.

To get it right you need the underlying observations or a mergeable structure like a histogram or t-digest, which is exactly why monitoring systems store those rather than pre-computed percentiles. If a dashboard is showing "average p99", it's showing a number that doesn't correspond to anything real. I'd also flag the related trap: if a page makes twenty backend calls, roughly 18% of page loads hit a p99 event on at least one call, so the user-facing tail is much worse than the per-service one.

3. Explain a p-value to a product manager.

It's the probability of seeing a difference at least this large if the change actually did nothing. Small p-value means the data would be surprising under the assumption that nothing is happening.

The thing I'd be careful to say is what it isn't. p = 0.03 does not mean a 97% chance the effect is real — it's computed assuming no effect, so it can't also tell you the probability that assumption is false. And it says nothing about size: with our traffic volume a completely worthless 0.02% difference will eventually hit p < 0.001. So I'd report the absolute difference, the relative lift and a confidence interval, and treat the p-value as the least interesting number in the summary.

4. We compared two rerankers on 200 queries: 0.71 versus 0.73. Ship it?

Not on that evidence. At n = 200 the 95% interval on a metric around 0.72 is roughly ±6 percentage points, so those two intervals almost completely overlap. A formal test gives about p = 0.66, which is no evidence of a difference at all.

To detect a gap that small at 80% power you'd need on the order of 7,900 queries per arm. But before collecting them I'd check one thing: if both models were scored on the same queries, the comparison should be paired, which removes query-to-query difficulty variation. That can be the difference between seeing nothing and seeing a clear effect — in a case I worked through, the same data went from p = 0.34 unpaired to p = 0.000003 paired.

5. What's the difference between standard deviation and standard error?

Standard deviation describes how spread out the data is — a property of the population that doesn't shrink when you collect more. Standard error describes how uncertain you are about the mean, and it's the standard deviation divided by the square root of n, so it does shrink.

The practical consequence is the square root. Halving your uncertainty needs four times the data; a tenfold improvement needs a hundredfold. That's why experiments take as long as they do, why detecting small effects is so expensive, and why techniques that reduce variance directly — like using pre-period data as a covariate — are worth real engineering effort. They buy precision without buying traffic.

6. Our fraud model is 99% accurate. Good model?

Probably a useless one, and accuracy is the wrong metric. If fraud is 1% of transactions, a model that predicts "not fraud" every single time is also 99% accurate and catches nothing.

I'd ask for precision and recall at the operating threshold, and PR-AUC rather than ROC-AUC, because ROC-AUC looks flattering on imbalanced data. Then the base rate question: even a genuinely good detector — 95% recall, 1% false positive rate — run against a 0.1% event has precision under 9%. Ten false alarms for every real catch, which means the review team stops trusting it within a week. For rare events the lever that matters is the false positive rate, not recall: dropping FPR from 1% to 0.1% takes precision from about 9% to about 49%.

7. The team checks the experiment dashboard every morning and stops when it goes green. What's wrong?

That's peeking, and it inflates the false positive rate badly. Every check is another opportunity to cross the threshold by chance, so the 5% error rate you think you have isn't the one you actually have.

I simulated this against a true null: one check gives about 4%, but twenty checks gives about 25%. Three weeks of daily monitoring means the experiment declares a winner a quarter of the time when the variant is literally identical to control. The fixes are either committing to a fixed sample size computed in advance, or using a method designed for continuous monitoring — sequential testing or group sequential boundaries — which spend the error budget deliberately across checks instead of pretending they didn't happen.

8. A variant wins on mobile and wins on desktop, but loses overall. How?

That's Simpson's paradox, and there's no arithmetic error — it happens when the traffic mix differs between arms and the segments have very different baseline rates.

Concretely: if desktop converts at 6% and mobile at 2%, and one arm is 80% desktop while the other is 90% mobile, the pooled comparison is mostly measuring the device split rather than the variant. The important part is that no statistical rule tells you which view is correct — that's a domain judgement. Here it signals broken randomisation, so I'd trust the segments and fix the assignment, because no analysis rescues a broken experiment. But if the variant itself had caused the traffic shift, the pooled number would be the honest one, since that shift is part of the effect.

9. How would you put error bars on NDCG@10?

Bootstrap it. NDCG doesn't have a convenient closed-form standard error, and I'd rather resample than go looking for one.

Resample the query set with replacement a couple of thousand times, recompute NDCG each time, and take the 2.5th and 97.5th percentiles of that distribution. Same technique works for MRR, medians, p99 and any ratio metric. Two caveats: resample whole queries, not individual judgements, because the query is the independent unit; and the bootstrap can't fix a biased sample — if the evaluation set is hand-written queries that don't look like production traffic, you'll get a very precise estimate of the wrong number.

10. We tested twelve metrics and one came out significant. Thoughts?

With twelve independent tests at α = 0.05, the chance of at least one false positive is about 46% even if the change did absolutely nothing. So one significant result out of twelve is roughly what I'd expect from noise alone.

What I'd want to know is whether that metric was nominated as the primary one before the experiment ran. If it was, it's a real result. If it was found by scanning the dashboard afterwards, it's a hypothesis, not a finding, and the right response is to run a confirmatory experiment on that single metric. Going forward I'd pre-register one primary metric plus a small number of guardrails, and apply Benjamini–Hochberg to anything exploratory rather than treating every metric as a candidate headline.

11. When would you use the median over the mean, and when is the mean actually right?

Median for anything I'm describing as a typical experience — latency, session length, order value — because those are all right-skewed and the mean gets dragged by the tail.

The mean is right when I care about a total, because then the outlier genuinely counts. Monthly token spend is the clearest case: one enormous request really does appear on the bill, so I want the mean multiplied by volume, not the median. Capacity planning is similar — total load is a sum. The rule I use is "mean for things you sum, median for things you experience", and if the two differ a lot that gap is itself worth reporting, because it tells you the distribution is skewed.

12. How much data do we need for this experiment?

I can't answer without one thing from the product side: the smallest effect that would actually change our decision. Sample size falls out of that, the baseline rate, and the power and significance levels we want.

The relationship worth knowing is that required sample scales with roughly the inverse square of the effect size — so detecting a 5% relative lift needs about four times the traffic of a 10% one. For a 3% baseline conversion and a 10% relative lift at 80% power, it's around 53,000 per arm, which at our volume is days rather than weeks. I'd compute this before launching, not after, because if the answer comes back as eleven weeks that changes what we choose to test.

Where this leaves you

You now have the tools to tell whether a number means anything: how it was sampled, how uncertain it is, and what else could have produced it. That is the foundation the next two chapters stand on.

Chapter 18 uses it to evaluate models honestly — which is mostly an exercise in not fooling yourself about generalisation. Chapter 19 uses it to run experiments on live traffic, where the statistics are the same but the operational traps are entirely different.