A/B Testing and Experimentation

Because everything else changes at the same time. If revenue rose 4% the week we shipped, that week also had a marketing email, a competitor stock-out and a seasonal shift. Before-and-after can't separate our change from any of it.

A simultaneous control group experiences all those same external factors, so the difference between arms isolates our change. And the reason randomisation specifically matters is that it balances confounders we never thought to measure — device, tenure, market, habit. That's something no statistical adjustment can do, because adjustment only handles variables you identified.

No, for two separate reasons. First, that's peeking — if we check daily and stop when it looks good, the false positive rate isn't 5%. Simulated against a true null, twenty checks takes it to around 25%.

Second, day two is exactly when novelty inflates results. In a worked example the first three days showed +11% and it settled at +6.3%; shipping early would have promised nearly double what the change delivers. I'd want the planned duration, whole weeks, and the daily effect curve to confirm it's flat. If we genuinely need the option to stop early, that's a decision to make upfront — group sequential boundaries or always-valid intervals — not something to improvise.

Yes, and I'd stop the analysis. A chi-square test on that split gives p around 3e-14 — it is not chance. It looks like a trivial 2.4% imbalance, which is exactly why this needs to be an automated check rather than something you eyeball.

Sample ratio mismatch invalidates the whole experiment, because whatever is excluding users from one arm almost certainly correlates with behaviour, and it can't be corrected in analysis. I'd check the usual causes in order: does the treatment error out for some users, so they never log exposure; is there a redirect or latency difference; is bot filtering applied after assignment; and are both arms logging exposure at the same point in the flow. That last one is the most common and the most boring.

User by default, because it gives a consistent experience across visits and devices and matches how people actually perceive the product. Session or request give more power but let the same person see both variants, so they're only safe for changes nobody can notice — a caching or infrastructure test.

The thing I'd flag is that whatever I randomise on, I must analyse on. If I assign by user and then compute significance over sessions, one heavy user's twenty sessions get counted as twenty independent observations, which understates variance and produces confident nonsense. And if there's interference — shared inventory, a shared model retraining on live behaviour — user randomisation breaks down and I'd look at cluster or switchback designs.

No, and this is exactly why guardrails exist. Click-through is a proxy; revenue is closer to what we actually care about. If the proxy moved and the real metric didn't, the proxy has stopped tracking the thing it was standing in for.

The mechanism is usually explainable — a ranker that learned to promote visually appealing products gets more clicks and more returns. I'd check the return rate and margin specifically. More broadly this is Goodhart's law: optimisation pressure finds the cheapest way to satisfy a measure, which is rarely the thing the measure represented. The lesson I'd take forward is to validate a proxy against the real outcome at least once before building a roadmap on it.

First I'd confirm the arithmetic, because the relationship is inverse-square: halving the effect you want to detect roughly quadruples the traffic needed. So a 3% target is genuinely expensive compared to a 10% one.

Then the options in order of preference. Variance reduction — CUPED using each user's pre-period behaviour — is usually the best value; at a pre/post correlation of 0.76 it removed 57.6% of the variance, worth about 2.4 times the sample size, for the cost of a join. A more sensitive metric closer to the change helps. Testing a bolder variant with a bigger expected effect helps. Interleaving, if it's a ranking comparison. And if none of those work, the honest answer is to ship on judgement with monitoring — but not to run an underpowered test and interpret the result.

Both arms get an identical experience, so the platform should report no difference. It's testing the measurement system rather than a product change.

It catches broken assignment, instrumentation that differs between arms, and pre-existing bias. The most valuable thing it catches is understated variance: if I run many A/A tests, about 5% should come out significant at alpha 0.05, and if 15% do then my analysis is treating correlated observations as independent and every result I've shipped was overconfident. That's why one A/A test proves little — the value is in running them continuously and tracking the false positive rate over months.

You adjust each user's outcome by their own behaviour before the experiment started. A user who always spends heavily will spend heavily in either arm, so that predictable part is noise as far as measuring the treatment goes — subtract it and what's left is a much tighter estimate.

It's unbiased because pre-period data can't be affected by a treatment that hadn't happened yet. The variance reduction equals the squared correlation between pre and post periods, so r = 0.7 removes about half the variance — in a case I ran, r = 0.76 gave 57.6% reduction, equivalent to 2.36 times the traffic. It doesn't help for new users with no history, and it reduces variance rather than bias, so it can't rescue an experiment with sample ratio mismatch or interference.

First, which kind of flat. If the experiment was adequately powered and the interval is tight around zero, that's a real finding: the change doesn't work, and I'd remove the code rather than leave a dormant variant as permanent complexity.

If the interval is wide and includes effects we'd have shipped for, it's inconclusive rather than negative, and the answer is more data or variance reduction. I'd also check for dilution — if only a fraction of assigned users could actually experience the change, a real effect gets averaged away, and re-running with assignment at eligibility often fixes it. The cultural point I'd make is that null results need to be recorded as findings with their intervals. A team that treats them as embarrassing will go looking through segments for something to report, and it will find something.

Mostly like anything else, with five specific complications. I'd pin the model version for the whole run, because a provider updating the model mid-experiment silently changes the treatment. I'd fix temperature to reduce variance that has nothing to do with my change.

Cost per session becomes a first-class guardrail, not a footnote — a conversion lift that costs more in inference than it earns isn't a win, and that calculation should be in the brief. Quality can't be scored directly, so I'd use behavioural proxies like task completion and escalation rate, backed by offline judging on a sample. And latency is asymmetric, since generation takes seconds, so I'd stream the response and watch time-to-first-token, otherwise I'm testing the feature plus a slowdown. Above all I'd do the offline evaluation first — rejecting a bad prompt in ten minutes beats spending two weeks of traffic on it.

Because individual wins don't compose. They overlap and interact, some were novelty that decayed after launch, and at a 5% false positive rate some were never real. Each was measured against a control that already contained the previous changes, so the effects aren't independent increments.

The instrument for answering this properly is a long-run holdout — a small slice, typically 1 to 5% of users, kept on the old experience for a quarter. It measures what the accumulated shipping actually delivered together. It's the least popular measurement in any organisation, because it sometimes shows a quarter of work was worth less than the sum of its reported wins, which is exactly why it's worth having.

Interleaving for ranking comparisons. You blend both rankers' results into one list and see which source gets clicked, so every user is their own control — the same reason a paired test beats an unpaired one. It's roughly an order of magnitude more sensitive, which matters because between-user variance normally swamps ranking differences. The limitation is it tells you which ranking users prefer, not what happens to revenue.

A bandit when I want the best outcome during a short campaign with several options — it shifts traffic toward whatever is winning. But I'd be explicit that it optimises rather than measures: it deliberately starves losing arms, so I end up with a vague estimate for everything except the winner, and the adaptive allocation breaks standard confidence intervals. If I need a reliable effect size for a lasting decision, that's a fixed A/B.