Analysis cannot rescue a badly designed experiment. Every meaningful decision — what you randomise, who counts, how long you run, what would make you stop — is made before a single visitor is bucketed, and none of it can be fixed afterwards.
That is the whole argument for taking design seriously. An hour spent on design saves a fortnight of arguing about a result nobody trusts.
This expands the experiment design section of the cheat sheet.
The randomisation unit is the first decision
The unit you assign to arms — visitor, logged-in user, session, or account — has to satisfy two rules:
- It must match the analysis unit. If you randomise by user, you analyse per user.
- It must be stable for the whole experiment. A unit that changes arms mid-journey destroys the comparison.
Session-level randomisation is the classic trap. It looks harmless, and for a single-page change it often is. But on any journey spanning multiple visits — considered purchases, B2B signups, anything with a delivery decision — a visitor can see control on Monday and variant on Wednesday. Their eventual purchase is then attributable to neither, and your effect washes out toward zero.
Where individuals interact with each other, the unit has to go up a level. Shared inventory, marketplace supply, social feeds, or a shared team workspace all mean one unit’s treatment affects another’s outcome. That breaks the independence assumption underlying every standard test. The fixes are cluster randomisation (assign whole accounts, stores or geos) or switchback designs (flip the whole system between arms in time slices), and in both cases the analysis unit must be the cluster, or your intervals come out far too narrow.
Assignment itself should be a deterministic hash of the unit ID plus an experiment-specific salt. That gives you stable, reproducible allocation, and independence between concurrent experiments.
Triggering: count only who could be affected
If you change something on the returns policy page, and you measure every visitor to the site, you have diluted a real effect with a large majority who never saw it.
Trigger-based analysis counts only units that reached the point where the change applies. Done properly it dramatically increases sensitivity — often the difference between a test you can afford and one you cannot.
The subtlety: the trigger must be defined identically in both arms. If your variant fires an exposure event when a component renders, and the control has no such component, you will only capture triggered users in one arm — and you have built a broken comparison rather than a sensitive one. Instrument the trigger at the point of eligibility, before the branch, in both arms.
Related, and worth stating: analyse by assigned arm, not by whether the treatment actually rendered. That is intention-to-treat. Filtering to users who “properly received” the variant reintroduces selection bias, because whatever caused the failure to render is probably correlated with something else about those users.
Sample size and runtime
Compute the sample size before launch. For proportions at α = 0.05 two-sided and 80% power:
n ≈ 16 · p(1−p) / Δ² per armThen convert to days using your actual eligible traffic, and round up to whole weeks.
Always run complete weekly cycles, minimum one and usually two, even if you hit the sample target on day four. Weekday and weekend visitors differ in intent, device, and basket size. Stopping mid-week bakes in whatever mix you happened to catch.
If the required runtime is absurd — and it often is on lower-traffic sites — you have four honest options, in order of preference:
- Reduce variance (see CUPED below) — free precision without more traffic
- Test a bigger change — small tweaks need enormous samples; if you can only afford to detect a 10% lift, test something that might plausibly deliver one
- Move upstream to a more sensitive metric — with a clear-eyed understanding that you are now measuring a proxy
- Accept a lower confidence level and say so explicitly
The dishonest fifth option is running underpowered and reading the result anyway. That is how organisations accumulate a portfolio of “wins” that never show up in revenue.
Guardrails and the OEC
Before launch, agree:
- One primary metric that decides the test. One. If two metrics can decide it, you have doubled your false-positive rate and given yourself room to pick whichever looks better.
- Secondary metrics for understanding mechanism, read descriptively, without significance claims unless you correct for multiplicity.
- Guardrail metrics that must not degrade even if the primary wins: page latency, error rate, refund rate, unsubscribes, support contacts, margin.
Guardrails are what stop a “winning” test from being a slow-motion loss. A checkout redesign that lifts conversion 3% while raising refunds 8% is not a win, and without a refunds guardrail nobody finds out for a quarter.
The OEC — overall evaluation criterion — is the agreed answer to “what are we optimising?”. Sometimes it is a single metric; sometimes it is an explicit trade, like revenue weighted against a retention proxy. Writing it down before the test removes the post-hoc argument about which number mattered.
Variance reduction: precision without traffic
CUPED is the highest-leverage technique available to most teams, and it is under-used because the name sounds harder than the idea.
The intuition: some users are simply higher spenders than others, and that variation has nothing to do with your treatment. If you know how each user behaved before the experiment, you can subtract that predictable component and compare what is left.
Y′ = Y − θ(X − X̄) where θ = cov(Y,X) / var(X)X is a pre-period covariate — typically the same metric measured over the weeks before the test. The variance reduction is 1 − ρ², where ρ is the correlation between pre and post behaviour. A correlation of 0.6 cuts variance by 36%, which cuts required runtime by roughly the same proportion.
It works best for logged-in products with returning users and pre-period history. It does nothing for brand-new visitors, since there is no pre-period to use.
Stratification is the cheaper cousin: randomise within strata such as device, country, or new-versus-returning, so the arms are balanced on known drivers rather than balanced only in expectation. It reduces variance and makes segment-level reads more trustworthy.
Stopping rules: fixed, sequential, or Bayesian
You must choose one before you start.
Fixed horizon. Set the sample size, do not look until you reach it. Statistically clean, organisationally difficult, and the assumption behind every textbook p-value.
Sequential testing. Designs built for continuous monitoring — mSPRT, group-sequential with alpha spending, always-valid confidence sequences. These let you look whenever you like without inflating the false-positive rate, at the cost of needing a somewhat larger sample for the same power. For most commercial teams this is the right default, because people are going to look at the dashboard whether or not you tell them to.
Bayesian. Report the posterior: probability the variant beats control, and expected loss from shipping the wrong arm. Stop when expected loss falls below a threshold you set in advance. This answers the business question directly. The prior is a genuine modelling assumption — with a weak prior and reasonable volume, results converge to the frequentist answer, so the choice is more about communication than about truth.
What you cannot do is run a fixed-horizon design and monitor it daily, stopping when it goes green. That is peeking, and it can push a nominal 5% false-positive rate above 30%.
When a bandit is the wrong tool
Multi-armed bandits adaptively shift traffic toward whichever arm is performing best. That is exactly right when your goal is to minimise regret during a short-lived allocation decision — which creative to serve on a three-day promotion, say.
It is exactly wrong when your goal is a clean, reusable estimate of an effect. Bandits deliberately unbalance allocation, which makes the causal estimate harder to interpret, entangles it with time trends, and leaves you unable to say confidently how much the winner was worth. If someone will ask “how much did that change deliver?” six months later, run an experiment, not a bandit.
Ship it server-side if you can
Client-side testing swaps content after the page renders. That produces flicker — the control briefly visible before the variant appears — which is visible to users, biases the variant downward, and is worst on exactly the slow connections where conversion is already fragile.
Server-side or edge assignment with feature flags removes flicker entirely, supports staged rollout, and gives you an instant kill switch. It costs more engineering. On any high-traffic surface it repays that quickly, both in data quality and in not showing customers a flash of the wrong page.
The pre-launch checklist
| # | Check | Failure if skipped |
|---|---|---|
| 1 | Hypothesis, primary metric and MDE written down | Result reinterpreted to fit |
| 2 | Sample size and end date fixed in advance | Peeking, inflated false positives |
| 3 | Randomisation unit equals analysis unit | Invalid comparison |
| 4 | Trigger scoped to affected users, both arms | Diluted effect, underpowered |
| 5 | Guardrails defined | A win that costs money elsewhere |
| 6 | Both arms QA’d across device, browser, logged-in state | Broken variant read as a loss |
| 7 | Outlier and refund rules set | Post-hoc rule choice = p-hacking |
| 8 | Sample ratio mismatch check automated | Days of unusable data |
| 9 | Full weekly cycles | Day-of-week composition bias |
| 10 | Reportable segments pre-registered | Segment mining |
Where this connects
- The statistics that make the readout valid are in hypothesis testing
- The ways a well-designed test still goes wrong: pitfalls and validity threats
- When randomisation is impossible: causal inference without an A/B test
- Definitions in one line: the cheat sheet