The uncomfortable truth about online experimentation is that a broken test does not look broken. It produces a number, a confidence interval, a green tick and a slide. Nothing in the output announces that the result is an artefact.

This is the catalogue of ways that happens, and what each one costs. It expands the pitfalls and validity threats section of the cheat sheet.

Peeking: the expensive habit

You set the test to run for two weeks. On day three the variant is up 8% with p = 0.04. Someone calls it and ships.

The problem is that a p-value’s guarantee applies to a single test at a predetermined sample size. Every additional look is another chance for random fluctuation to cross the threshold. Check daily for a fortnight and your effective false-positive rate is not 5% — depending on the checking pattern it can climb past 30%.

Worse, early results are biased in a specific direction. Random walks that wander far from zero early are precisely the ones that cross the significance line first, so a test stopped early on a big number has been selected for exaggeration.

The fixes, in order of practicality:

  1. Adopt sequential testing — mSPRT, group-sequential with alpha spending, or always-valid confidence sequences. Look as often as you like; the maths accounts for it.
  2. Use a Bayesian readout with an expected-loss stopping rule agreed in advance.
  3. Fix the sample size and genuinely do not look. Rarely survives contact with a real organisation.

What does not work is fixing the horizon and monitoring daily “just to check nothing is broken”, then acting on what you see. Monitoring for breakage is legitimate; monitoring for wins is peeking.

Multiple comparisons

Every extra comparison is another lottery ticket. At α = 0.05, twenty independent comparisons yield roughly one false winner by chance alone:

P(at least one false positive) = 1 − 0.95^m

Five metrics across four segments is twenty comparisons. Three variants against control on six metrics is eighteen. Teams run this every week without noticing.

Corrections:

  • Bonferroni — test each at α/m. Simple, strict, costs power.
  • Benjamini–Hochberg — rank the p-values, compare each to (i/m)·q, control the expected proportion of false discoveries. The practical choice when scanning many metrics, because it does not flatten your power the way Bonferroni does.

The organisational fix is cheaper than either: one primary metric, declared in advance. Secondaries are read descriptively for mechanism, not for decisions.

Full treatment — where multiplicity hides, and the corrections worth using — in the multiple comparisons problem.

Sample ratio mismatch: stop, do not analyse

You planned a 50/50 split. You observe 52/48 across 200,000 users. That sounds close. It is not — at that volume the deviation is wildly improbable, and a chi-square test against expected allocation returns a p-value in the region of 10⁻¹⁵.

SRM means the randomisation or the logging is broken. Common causes:

  • The variant’s tracking fires later, so some users are lost before being recorded
  • A redirect drops a browser segment
  • Bot filtering applied unevenly across arms
  • Users reassigned mid-test by a caching or hashing change

The rule is unambiguous: an SRM test is not salvageable by analysis. If assignment is broken, there is no valid comparison to rescue. Fix the cause and rerun. Flag at p < 0.001 and automate the check from day one — it should be the first thing on every results page.

Simpson’s paradox

The variant wins on desktop. The variant wins on mobile. The variant loses overall.

This is not a contradiction, it is a mix effect. If the arms have different proportions of desktop and mobile users — which SRM checks might not catch if the overall split is fine — the pooled number is a weighted average with different weights per arm.

It also appears when traffic mix shifts mid-test: a campaign starts on day five and pours in low-intent mobile traffic. Both arms get it, but if it lands unevenly in time and your arms have different exposure patterns, the pooled comparison drifts.

Defences: stratified randomisation on the big mix variables, checking that arm composition matches on device and geography, and looking at pre-registered segments as a diagnostic before pooling.

Novelty and primacy

Two opposite biases, both time-dependent, both invisible in a single-point readout.

Novelty: returning users engage with something because it is new, not because it is better. Shows as a strong early lift that decays. Ship it and the effect evaporates within weeks.

Primacy: regular users are disrupted by a change and perform worse until they relearn the interface. Shows as an early loss that recovers. Kill it on day three and you have killed a winner.

Both are detected the same way: plot the effect by cohort week rather than reporting one pooled number, and check new users separately from returning ones. If the effect among new users — who have no prior expectation — is stable while returning users show a decaying or recovering curve, you have your answer.

This is also the argument for a holdback: keep a small slice on control for weeks after rollout and confirm the day-14 effect is still there at day 90. It usually is not, entirely.

Regression to the mean

Extreme observations drift back toward average on remeasurement, with no intervention at all.

The practical consequence: if you pick the pages that performed worst last month and “optimise” them, they will improve. So will the ones you did not touch. Any before-and-after comparison on a selected-for-extremeness set will show improvement that is purely statistical.

This is one of the strongest arguments for a control group in absolutely everything, including work that feels too obvious to test.

p-hacking and HARKing

Rarely deliberate. Almost always what happened when a flat test became a win.

p-hacking is trying variations of the analysis until something clears 0.05 — different outlier rules, different date windows, excluding a “weird” day, switching denominators, adding a filter. Each individual choice is defensible. The sequence is not, because you chose among them by looking at the results.

HARKing — Hypothesising After the Results are Known — is presenting a post-hoc explanation as if it had been the original prediction. It turns exploratory noise into a confirmed finding, and it is contagious, because the story is always plausible.

Segment mining is the specific version everyone does: a flat test “wins on mobile in Germany for returning users”. With enough segments, something always wins. Post-hoc segments are hypotheses for the next test, never conclusions from this one.

The defence is pre-registration, and it does not need a process document — a dated note in the ticket stating hypothesis, primary metric, MDE, exclusion rules and the segments you will report is enough. Five minutes, and it makes the difference between analysis and archaeology.

Instrumentation and contamination

Flicker. Client-side variants render after the control is briefly visible. Users see it, it biases the variant downward, and it is worst on slow connections.

Instrumentation bias. The variant carries an extra script or a changed selector, so its tracking fires differently. The measured difference is then partly measurement rather than behaviour.

Cross-contamination. Shared devices, cleared cookies, cached pages, CDN mixing — all put users in both arms. Every instance dilutes the true effect toward zero, so contaminated tests systematically under-report.

Network interference. One unit’s treatment affects another’s outcome through shared inventory, marketplace supply or social feeds. Breaks the independence assumption entirely; needs cluster or switchback designs.

Winner’s curse

Effects that just cleared the significance bar are biased upward. This is not a flaw in anyone’s analysis — it is selection. Only sampled effects large enough to reach the threshold get through, so among all results at the margin, the true effects are systematically smaller than measured.

Expect measured lifts to shrink on rollout, particularly from underpowered tests, and particularly from tests stopped early. If your programme reports +5% wins that never appear in the revenue line, this plus peeking explains most of it.

The mitigation is unglamorous: adequate power, holdbacks that measure the effect again after rollout, and a healthy scepticism about any result whose confidence interval barely excludes zero.

Twyman’s law

Any figure that looks interesting or unusual is usually wrong.

A +40% lift on a checkout tweak is a tracking bug until proven otherwise. So is a metric that doubles overnight. The instinct to celebrate is exactly backwards — the bigger the number, the harder you should look at the instrumentation first.

Where this connects