The Multiple Comparisons Problem: How Slicing an A/B Test Manufactures Winners

What the multiple comparisons problem actually is

A significance threshold is a rate, not a verdict. Setting α = 0.05 means: in a world where the variant does nothing at all, you will still declare a winner 5% of the time. That error rate is the price of admission, and it is tolerable — for one comparison.

Run more than one and the errors accumulate. For m independent comparisons, the probability of at least one false positive somewhere in the set is:

FWER = 1 − (1 − α)^m

That quantity is the family-wise error rate, almost always written FWER. The “family” is the whole set of comparisons you performed; the FWER is the probability that at least one of the winners you declare from that set is false.

It is worth being precise about how that differs from α, because the two get conflated constantly:

• α — the per-comparison error rate. The chance that one particular test, considered in isolation, falsely declares a winner. This is the 5% everyone quotes.
• FWER — the family-wise error rate. The chance that anything at all in the batch is a false winner. This is the number that describes your actual exposure, and nobody quotes it.

Run a single comparison and the two are identical. Run more than one and they diverge — fast.

Comparisons	Chance of ≥1 false winner at α = 0.05
1	5%
3	14%
5	23%
10	40%
20	64%
40	87%
100	99.4%

At twenty comparisons you are more likely than not to be looking at at least one thing that isn’t real. At forty — which is five metrics across eight segments, a perfectly ordinary weekly read — a false positive is close to guaranteed.

The critical detail, and the one that makes this so hard to police: the false positive doesn’t announce itself. It has the same p-value, the same confidence interval and the same green tick as a real effect. You cannot tell them apart by looking. The only defence is knowing how many comparisons produced the one you are looking at.

The count that matters is not how many comparisons you reported. It is how many you could have reported and would have written up had they won.

That second framing — the garden of forking paths — is the one people resist. If you would have taken “wins on Android” as a finding, then Android was in the family, whether or not you happened to look at it. Analysis flexibility is multiplicity even when it never produces a single extra line of code.

Where it hides in an A/B test

Multiple metrics

The most common source, because it feels like diligence rather than fishing. A typical readout carries conversion rate, revenue per visitor, average order value, add-to-cart rate, bounce rate, pages per session and checkout completion. Seven metrics against control is seven comparisons: a 30% chance of at least one spurious mover before anyone touches a segment.

Metrics also correlate, which cuts both ways. Correlation means the true error rate is lower than the independent-case formula suggests — but it also means the corrections that assume independence are wrong, and, more importantly, that correlated metrics moving together is not the corroboration people read it as. Add-to-cart rate and conversion rate rising together is close to one fact, not two.

The subtler version is metric redefinition. Conversion rate per session, per user, per new user, excluding bounces, on a 7-day window versus 14. Each variation is another comparison. When someone says “it’s significant if you look at it per user”, they have described a multiplicity problem, not a finding.

Segmentation and breakdowns

This is where the damage is worst, for three reasons that compound.

The comparison count explodes multiplicatively. Device (3) × new-versus-returning (2) × traffic source (4) is 24 sub-populations before you have added country or landing page. Cross them with five metrics and you are at 120 comparisons in a single analysis session.

Segments are underpowered, so their results are noisier and more extreme. MDE scales with 1/√n. A segment holding 15% of your traffic has roughly √(1/0.15) ≈ 2.6× the minimum detectable effect of the full test. If the test as a whole was powered to detect +3%, that segment can only reliably detect around +8%. Any segment result smaller than that which nonetheless reads as significant is, by construction, being carried by variance.

The selected segment’s effect is systematically inflated. You did not pick a segment at random — you picked the one with the biggest number. Conditioning on “won” selects for upward noise, so the reported +12% is a biased estimate of a true effect that is much smaller, or zero. Even when the segment effect is genuinely real, the size you report will not replicate. This is the winner’s curse operating inside a single test.

Put together: post-hoc segment findings are the least reliable output an experimentation programme produces, and they are the output most likely to become a strategy.

Multiple variants

An A/B/C/D test is three comparisons against control, not one — and if anyone compares the arms to each other, six. Two costs land at once: the family-wise error rate rises, and each arm gets a quarter of the traffic rather than half, so per-arm MDE goes up by about 40%. More arms means more chances to be fooled and less ability to detect anything real.

There is a genuine efficiency argument for multi-arm tests — shared control, one setup, one allocation period — but it holds only if the arms are pre-specified and the analysis uses a method built for many-versus-one comparison (see Dunnett, under Fix the stats) rather than four independent t-tests.

Looking repeatedly over time

Peeking is multiplicity along the time axis. Ten daily checks of the same metric is ten comparisons of a sequentially-updating statistic, which is why unmonitored daily checking pushes an advertised 5% error rate past 30%. It is covered properly in pitfalls and validity threats and when you can stop a test, but it belongs on this list — teams that have fixed their metric count often have not noticed they are still multiplying by twenty looks.

Across the programme, not just the test

Run 60 tests a year at α = 0.05, where most interventions genuinely do nothing, and you should expect roughly three “winners” that are pure chance — permanently written into the roadmap as learnings. This one has no per-test fix. It is why replication and holdback groups matter, and why a win rate that looks suspiciously like your α should prompt questions.

Solution part one: make the experiment simpler

Correction formulas are the second line of defence. The first is not generating the comparisons at all — and that is almost always the higher-return intervention, because it costs statistical power nothing.

Declare one primary metric. Not a shortlist. One number, chosen before launch, that the ship/no-ship decision rests on. Everything else is read descriptively for mechanism, explicitly labelled as such, and never carries a significance claim. This single habit removes most real-world multiplicity at zero cost.

Build an OEC if one metric genuinely is not enough. An overall evaluation criterion combines the things you care about into a single decision metric with the weights agreed in advance — revenue per visitor with a returns adjustment, say. Arguing about the weights beforehand is productive. Arguing about which of seven metrics counts, after the results are in, is not.

Pre-register your segments, and cap them at three. Write them in the ticket before launch, with the reason you expect heterogeneity there. Three pre-registered segments is a manageable family. Eight discovered ones is archaeology. And if a segment matters enough to decide on, power the test for the segment, not for the total.

Treat post-hoc segments as hypothesis generation, full stop. The correct output of “it won on mobile” is a mobile-only test, not a mobile rollout. Say so in the write-up in those words — the discipline holds far better when the finding gets a next step rather than a dismissal.

Cut arms. Two well-powered variants beat five underpowered ones nearly every time. If you have five ideas, the ranking exercise is the work.

Fix the analysis rules before launch. Outlier caps, exclusion windows, the metric denominator, the run length. Each rule left open until after the data arrives is a fork in the path, and forks multiply exactly like metrics do.

Buy power rather than leniency. Variance reduction — CUPED, stratified randomisation, capping heavy tails — makes every comparison sharper, which is what makes a correction affordable rather than fatal. A result that is only just significant uncorrected cannot survive any correction at all; the answer to that is more power, not fewer corrections.

Replicate anything expensive. A holdback group, or simply running the winner again, is the most honest multiplicity correction available and needs no statistics at all.

Solution part two: correct the statistics

When comparisons are genuinely unavoidable, pick the method that matches what you are protecting against. Two abbreviations run through the Controls column below, and the choice between them is the real decision:

• FWER — family-wise error rate. The probability that at least one of your declared winners is false, across the whole family of comparisons. A strict, all-or-nothing guarantee.
• FDR — false discovery rate. The expected proportion of your declared winners that are false. Looser, and deliberately so.

Controlling FWER at 5% says “there is a 5% chance anything here is spurious”. Controlling FDR at 10% says “expect about one in ten of these to be duds”. Very different promises, and the second is far cheaper in statistical power.

Method	Controls	Rule	Use when
Bonferroni	FWER	Test each at α/m	Few comparisons, high cost of any false positive
Holm–Bonferroni	FWER	Sort p ascending; compare p₍ᵢ₎ to α/(m−i+1); stop at the first failure	Default replacement for Bonferroni — strictly more powerful, same guarantee
Šidák	FWER	Test each at 1−(1−α)^(1/m)	Comparisons genuinely independent
Dunnett	FWER	Purpose-built many-arms-versus-one-control	A/B/n tests — exploits the shared control, so it beats Bonferroni
Tukey HSD	FWER	All pairwise comparisons	You really do need every arm compared to every other
Benjamini–Hochberg	FDR	Sort p ascending; find the largest i with p₍ᵢ₎ ≤ (i/m)·q; reject all up to it	Scanning many metrics or segments for leads
Fixed-sequence gatekeeping	FWER	Order the hypotheses in advance; test at full α until one fails	You can rank importance beforehand — costs no power at all
Bayesian hierarchical model	Shrinkage	Partial pooling of segment effects toward the global effect	Segment-level reads you intend to do routinely

A few of those deserve elaboration.

Holm should be your default, not Bonferroni. It controls the family-wise error rate under exactly the same conditions — including arbitrary dependence between the tests — and it rejects at least everything Bonferroni does, usually more. There is no scenario where plain Bonferroni is preferable; it survives mainly because it is easy to do in your head.

Match the control to the consequence. If you are making one irreversible ship decision, you want FWER — the cost of a single false winner is the entire point of the exercise. If you are scanning thirty metrics to decide what to investigate next, FWER at m = 30 will flatten your power to nothing, and Benjamini–Hochberg at q = 0.1 is the sane trade: you accept that roughly one in ten leads is a dud in exchange for finding the other nine.

Fixed-sequence gatekeeping is the underrated one. Rank your hypotheses before launch: primary, then secondary, then tertiary. Test each at the full α, in order, and stop the moment one fails. Because the sequence is fixed in advance, this controls FWER with zero power penalty on the first hypothesis. Its cost is honesty: you must commit to the ordering before you see the data, and everything after the first failure is unreadable. That is often exactly the right shape for an experiment readout.

The Bayesian route sidesteps α adjustment entirely. Fit segment effects hierarchically and each segment’s estimate is pulled toward the global effect in proportion to how noisy it is. Small, noisy segments shrink hard toward the overall result; large, consistent ones move less. Extreme segment estimates stop appearing, because the model already encodes that an isolated +12% from 4% of your traffic is more likely to be noise than a genuine outlier. It is a better-behaved answer than “corrected or uncorrected” — see auto-optimisation and cohort Bayesian methods.

When not to correct

Corrections are not free, and applying them reflexively causes its own damage.

Guardrail metrics should not be corrected. Page load time, error rate, refund rate, unsubscribes — these exist to catch harm, and the cost asymmetry runs the other way. A false alarm costs an investigation; a missed regression costs customers. Leave them at full α, or run them one-sided as non-inferiority checks against a pre-agreed damage threshold.

Genuinely exploratory analysis does not need correcting — it needs labelling. Slice as widely as you like, provided the output is a ranked list of hypotheses for future tests and the write-up says so. The problem was never the slicing; it was the promotion of a slice to a conclusion.

A single pre-registered primary metric needs no correction, by definition. m = 1. This is why the organisational fix outranks the statistical one.

And correcting after the fact does not undo the fishing. Applying Benjamini–Hochberg to the twelve comparisons you chose to report, out of the sixty you actually looked at, is a correction with the wrong m — arithmetic decoration on a compromised analysis. The family has to be defined before you see the data, or the correction means nothing.