A test finishes. The dashboard shows +4.2%, p = 0.03, and a green tick. Someone says “95% confident the variant wins” and the change ships.
That sentence is wrong in a specific way, and the gap between what was said and what the maths supports is where a lot of money quietly disappears.
This is the deep dive behind the inference and hypothesis testing section of the cheat sheet.
What a p-value is, precisely
The p-value is the probability of observing a difference at least this extreme, assuming there is no real difference.
Read that twice, because the assumption clause is the whole thing. The p-value is computed in a world where the null hypothesis is true. It tells you how surprising your data would be in that world.
What it is not:
- Not the probability the variant is better
- Not the probability you are wrong
- Not the probability the result will replicate
- Not a measure of how big the effect is
p = 0.03 means: if the two variants were genuinely identical, data this extreme would show up about 3% of the time. That is meaningful evidence. It is not a 97% chance of being right.
The probability that the variant is better is a different quantity, and it requires a prior — which is exactly what Bayesian methods provide, and why “probability to beat baseline” is such a natural thing for stakeholders to hear.
The two errors, and the one nobody counts
| No real effect | Real effect exists | |
|---|---|---|
| You ship | Type I error — rate α | Correct |
| You don’t ship | Correct | Type II error — rate β |
Type I is shipping something that does nothing. It is the error the whole apparatus of significance testing is designed to control, at rate α — conventionally 5%.
Type II is failing to detect something real. Its rate is β, and 80% power means β = 20%: one in five genuine improvements gets binned as “flat”.
The asymmetry in how organisations treat these is stark. Type I errors get post-mortems. Type II errors are invisible — nobody writes up the winner they discarded. Over a couple of years of underpowered testing, an organisation can systematically train itself to believe nothing works, when in fact it has been throwing away every effect smaller than its detection threshold.
Choose α for the cost of the decision, not by convention. A risky platform migration deserves 0.01. A headline copy change can live happily at 0.10, because the cost of a false positive is a line of text.
Power and MDE: decided before, not after
Power (1 − β) is the probability of detecting an effect of a given size, if it is real. Minimum detectable effect is the smallest true effect your test can reliably find, given α, power and sample size.
MDE ≈ (z_α/2 + z_β) · √(2σ²/n)For proportions, the quick approximation for sample size per arm at α = 0.05 two-sided and 80% power:
n ≈ 16 · p(1−p) / Δ²where Δ is the absolute difference you want to detect. At a 3% base rate, detecting a 0.3pp lift needs roughly 51,700 per arm. Detecting a 0.15pp lift needs about four times that — because the Δ is squared.
This is the single most useful calculation in experimentation, and doing it before launch is what separates a test from a hopeful gesture. The honest conversation it forces is: “we can detect a 10% relative lift in twelve days, or a 5% lift in seven weeks — which are we actually expecting?” If the MDE you can afford is larger than any plausible effect, the test cannot answer the question and should not be run.
Confidence intervals, and the sentence to stop saying
A 95% confidence interval is built so that, across many repeated experiments, 95% of such intervals would contain the true effect. It is a statement about the procedure, not about this particular interval.
The Bayesian credible interval is the one that means what everyone assumes: given the model and prior, a 95% probability the true value lies inside.
Whichever you use, report the interval, not just the point estimate. Its width is the real finding. Consider three results, all “not significant”:
+1.2%, CI [−0.3%, +2.7%]— plausibly a decent win, plausibly nothing. Underpowered. Inconclusive.+0.1%, CI [−0.4%, +0.6%]— genuinely flat. The effect, if any, is smaller than you care about. That is a real finding.+8.0%, CI [−1.0%, +17%]— wildly underpowered. Tells you almost nothing.
Reporting all three as “no significant difference” throws away most of the information. The second is a decision; the first and third are requests for more data.
This is the most valuable habit in this entire article: “not significant” and “no effect” are different findings.
Choosing the right test
| Metric | Default | When the default fails |
|---|---|---|
| Conversion rate | Two-proportion z-test | Under ~30 conversions per arm → Fisher’s exact |
| Revenue per visitor | Welch’s t-test | Heavy tail → bootstrap, or cap at p99 |
| Items per order (a ratio) | Delta method | Never a plain proportion test |
| Time on task | Mann–Whitney U | Report medians alongside |
| Three or more arms | Chi-square, then pairwise | Correct for multiplicity |
Three notes that catch people out:
Always use Welch’s t-test, not Student’s. Welch does not assume equal variances between arms, and treatment usually changes spread as well as centre. It costs nothing and removes an assumption you cannot verify.
Ratio metrics need the delta method. Revenue per user and items per order have a random numerator and a random denominator. Treating them like a simple proportion understates the variance, which narrows your intervals and manufactures significance. This is one of the most common silent errors in home-grown analysis.
Mann–Whitney tests dominance, not means. It is robust to skew, which is attractive. But it can tell you the variant’s distribution is stochastically higher while total revenue is flat — and revenue is what the business asked about. Use it as a supporting read, not the decision metric.
Statistical significance is not practical significance
With enough traffic, everything becomes significant. A 0.02% lift on a large site will clear p < 0.05 eventually, and it may be worth nothing at all once you count the engineering time to maintain the change.
Effect size, expressed in standard-deviation units (Cohen’s d) or simply in money, is independent of sample size and is what you should actually argue about. Decide your practical threshold before the test — alongside the MDE — and write it down. “We will ship if the lower bound of the confidence interval exceeds +0.5%” is a decision rule. “We will ship if it goes green” is not.
And be careful with absolute versus relative. 2.0% → 2.4% is +0.4 percentage points absolute, +20% relative. Power calculations consume the absolute figure; stakeholders hear the relative one. Label every number.
A defensible reporting template
For any test, report:
- Primary metric, with denominator stated
- Absolute and relative effect, both labelled
- Confidence interval on the primary
- Sample size achieved versus planned
- Guardrail metrics, with intervals
- The outlier and exclusion rules applied
- A recommendation with its reasoning — including “inconclusive, needs longer”
Where this connects
- Distribution shape and variance, which determine everything above, are in descriptive statistics
- Making the test valid in the first place is experiment design
- The ways significance gets manufactured are in pitfalls and validity threats
- Every definition in one line: the cheat sheet