Ask people who run large experimentation programmes how to judge one, and a lot of them will point at velocity: how many tests you run, and how quickly you learn. Booking.com and Microsoft built their programmes on it, and Jeff Bezos is widely quoted as saying Amazon’s success was a function of how many experiments it ran per year, per month, per week and per day.
But there’s a vocal counter-argument, and it’s persuasive enough to turn up regularly in conference talks and board decks. Volume is a vanity metric. Chasing numbers produces badly planned tests. Run fewer, better-thought-through experiments instead. It gets nods, because it sounds like the voice of someone who has grown out of something.
I’ve run more than 6,000 experiments over twenty-odd years, and I’m on the side of volume. Here’s why.
The critics do have a point. Volume can absolutely be gamed. Split one idea into five button colours and you’ve quintupled your velocity without learning anything. Nobody is defending that.
But their fix takes a real problem (volume being gamed) and applies the wrong remedy (stop valuing it). Volume, read properly, is the best single indicator of whether an experimentation programme is healthy. And the metric the quality camp usually reaches for instead, win rate, is far easier to game and far more damaging when it is.
There are three arguments, and they’re in the tabs. Read the one you disagree with most.
Every change is an experiment. Most just don’t record the result.
Start with what an experimentation programme is actually for. It isn’t there to produce winning tests. It’s there to stop the business shipping changes that make things worse, and to find out which changes make things better.
Now look at the denominator nobody puts on the dashboard: how many changes shipped to the site this quarter, and how many of them went through a test?
Most teams have never counted. When they do, the number is usually small. The tested changes are the carefully chosen ones: a checkout tweak here, a PDP layout there. Meanwhile the navigation redesign, the new filter logic, the homepage refresh, the promo banner, the rebuilt search, the “quick copy update” to 400 category pages and the new cookie banner all go out untested, because they weren’t “CRO work”. They were just work.
Every one of those was an experiment. It changed something for real customers, and it had an effect. The only difference is that nobody measured it.
What the base rates say about shipping blind
The best-known public numbers on this come from Ronny Kohavi and colleagues, who ran experimentation at Microsoft, Amazon and Airbnb and published what they saw. In Trustworthy Online Controlled Experiments they report that across Microsoft, roughly a third of ideas improved the metric they were designed to improve, a third did nothing, and a third made it worse. At Bing and Google the success rate was lower, around 10–20%. Booking.com, which runs one of the largest programmes anywhere, has been widely reported as having around 90% of its experiments fail to produce a positive result.
Those are organisations with very good product people. Their ideas aren’t worse than yours. If anything, their ideas get more scrutiny before they ship.
So take the generous Microsoft split and run a deliberately simple, illustrative year:
| Changes | Help | Do nothing | Hurt | |
|---|---|---|---|---|
| Shipped through a test | 15 | 5 (kept) | 5 | 5 (caught and killed) |
| Shipped blind | 85 | ~28 | ~28 | ~28 (live, unmeasured) |
The experimentation team reports five wins. Genuinely good work, properly measured.
In the same year, around 28 changes went out that made the site worse, and nobody knows which ones they were. They are still live. Some of them will stay live for years, because nothing ever flagged them.
The numbers are made up, but the shape isn’t. With 15% coverage, your programme isn’t steering the site. It’s a small number of guarded steps inside a random walk.
Untested changes don’t just miss wins. They undo them.
There’s a nastier version of this, and anyone who has run a programme for more than a couple of years has seen it.
You test a new checkout layout. It wins, cleanly, at a decent effect size. It ships, and the experimentation team rightly reports it.
In the same quarter, twenty other changes land on the same funnel, none of them tested. A reworded delivery message. A promo banner that nudges the basket button below the fold on smaller phones. A new trust badge. Payment methods reordered. A “helpful” upsell on the basket page. A slower product image carousel. Each one is small, sensible and signed off by someone competent.
If the base rates above hold, around a third of them make things slightly worse. None of them needs to be a disaster. Seven changes each costing a fraction of a percent is plenty to cancel a clean win. The test dashboard says +3%. The revenue line says flat. Nobody can explain the gap, because the only change anyone measured was the one that helped.
That is the coverage gap in its purest form. Low coverage doesn’t only mean you’re missing harm. It means your wins don’t compound, because whatever you proved is sitting in a funnel that everything you didn’t prove keeps changing around it.
The hook I’d put on a slide:
You don’t get to keep your wins if 20 other changes ship blind.
So why is volume the health metric?
Because volume is the only practical proxy for coverage.
You can’t test 60% of meaningful changes while running six tests a year. A team that has pushed its coverage up has to be running a lot of experiments. There’s no other way to get there. So when someone says “we ran 200 tests last year”, the useful follow-up isn’t “that sounds like too many”. It’s “out of how many changes?”
High volume on its own proves nothing. But low volume all but guarantees low coverage, and low coverage guarantees you’re shipping damage you can’t see.
Coverage is king. Volume is how you get there. Win rate is how you fool yourself.
”If you’re running that many, they can’t be well thought through”
This is the objection I hear most, and it’s worth taking seriously, because it sounds like rigour.
The argument goes: a good experiment starts with research, turns it into a hypothesis, prioritises it against everything else, and is carefully designed. That takes time. So a team running hundreds of tests must be skipping those steps. Volume is evidence of sloppiness.
The problem is the hidden assumption. It treats the experiment as the output of research. It isn’t. The experiment is the research. It’s the most reliable research method we have, because it’s the only one that measures what customers actually do when the thing is actually different.
Session replay, surveys, heuristic reviews and usability testing are all useful. I lean on them heavily, particularly on sites without the traffic to test much. But they produce hypotheses. The experiment is what turns a hypothesis into a finding.
There is only a point testing what you don’t know
Follow that thought one step further and the “fewer, better-planned tests” position starts to eat itself.
The information you get from a test depends on how uncertain you were going in. If you were 90% sure a change would win and it wins, you’ve learned very little. You spent a month of traffic confirming what you already believed. If you were genuinely 50/50, either result changes what you do next. That’s the test worth running.
Heavy up-front planning pushes teams toward the first kind. By the time an idea has survived research, a hypothesis document, a prioritisation framework and a stakeholder review, it’s usually something everybody in the room already expects to win. Programmes like that tend to report healthy win rates and learn remarkably little.
If you already know the answer, it isn’t an experiment. It’s a ceremony.
The most commercially important test result I know of in the public record makes the point. Kohavi and Stefan Thomke described it in Harvard Business Review: an engineer at Bing had a small idea about how ad headlines were displayed. It was judged low priority and sat in the backlog for over six months. When someone finally ran it (it was cheap to build), it lifted revenue by 12%, worth more than $100 million a year in the US alone. It was, by their account, the best revenue idea in Bing’s history.
No prioritisation framework would have picked it. Cheap, high-volume testing found it anyway.
Planning should scale with the cost of being wrong
None of this is an argument against thinking. It’s an argument for spending your thinking where it pays.
Heavy planning made sense when tests were expensive. If an experiment needed a development sprint, a QA cycle and a client-side tool that flickered and polluted the sample if you looked at it wrong, then every test slot was precious and rationing them was rational.
When a test costs an afternoon, the maths flips. A two-week prioritisation process that decides which of ten cheap ideas to test costs more than testing all ten. The planning becomes the bottleneck it was meant to protect.
What every test needs is short: a clear question, one primary metric declared before launch, enough traffic to detect an effect worth having, and a decision rule you’ll actually follow. That’s rigour. A twelve-slide hypothesis deck is paperwork. Well-planned and numerous are not opposites.
When volume really is the wrong goal
To be fair to the other side, there are programmes where chasing volume is a mistake, and it’s worth naming them.
- Volume theatre. One idea split into four tests to hit a quarterly target. Button colours. Tests nobody intends to act on. If a test can’t change a decision, it doesn’t count, however many of them you run.
- Underpowered volume. Twenty tests that each run for a week on a page with 3,000 visitors aren’t twenty results. They’re twenty coin tosses with confidence intervals attached.
- Low-traffic sites. If your site can only power six tests a year, test capacity is your scarcest resource. You should ship the obvious fixes without testing them and spend your slots on genuinely uncertain decisions. There’s a whole article on that, and it’s the one case where I’d tell you to care about hit rate over volume.
What these have in common is that they count tests, not decisions under uncertainty. Volume is a health indicator in the same way a resting heart rate is. It tells you something real, and it’s pointless as a target in its own right. The fix for gamed volume is to count the right unit, not to stop counting.
What replaced volume, and why it’s worse
The usual alternative from the anti-volume camp is win rate: the percentage of experiments that produce a statistically significant improvement. Plenty of programmes report it, sometimes instead of velocity and sometimes alongside it as the “real” measure. It feels like the grown-up metric. It measures quality, not just activity.
It’s a trap, and the mechanism is plain Goodhart’s law. Make win rate the measure of a good programme and people will find ways to raise it, and almost all of them make the programme worse.
How to raise your win rate without getting better at anything
None of these requires bad intent. Most happen naturally once a team knows it’s judged on wins.
Test the safe ideas. The fastest route to a high win rate is to stop testing anything you might lose. Bold redesigns, pricing changes, removing a feature people assume is load-bearing: these are the tests with the most information in them, and they’re the first to go. What’s left is fixing obvious friction, which wins, and which frankly didn’t need testing.
Stop at the first good-looking day. Checking the dashboard until it crosses p < 0.05 and calling it is the most common way teams ship false positives. Under a win-rate target, peeking stops being a bad habit and becomes the job.
Slice until something wins. A flat test, the segment dropdown, and four clicks later it’s “significant for returning mobile users in the UK”. That’s the multiple comparisons problem, and a win-rate culture turns it into a standard operating procedure.
Move the goalposts. Conversion didn’t move, but add-to-basket did. Revenue per visitor is flat, but it’s “significant per user”. Each redefinition is another lottery ticket.
Invent the “directional win”. Not significant, but pointing the right way, so ship it and count it. This is where win rates of 60% come from.
Every one of those produces a better-looking number and a worse programme. Worse still, the wins it produces are inflated. Effects that only just clear the bar are systematically overestimated. That’s the winner’s curse, and a programme that selects for marginal wins is selecting for it deliberately. It’s why so many roadmaps full of “+4% winners” never show up in the revenue line.
A 90% win rate doesn’t mean you’re good at testing. It means you’re good at choosing tests.
Losing tests are the programme working
This is the part of the win-rate framing I find most corrosive: it tells everyone that a losing test is a failure.
Go back to the base rates in the Coverage Gap tab. If roughly a third of well-intentioned changes make things worse, then a programme that runs lots of tests should have lots of losers. Every one of them is a change that would otherwise have shipped and quietly cost money, possibly for years.
A losing test isn’t a failure. It’s a bug you caught before your customers did.
A programme with a low win rate and high coverage is catching harm at scale. A programme with a high win rate and low coverage is congratulating itself on a handful of safe bets while the damage ships around it. On a win-rate dashboard the second one looks better, and it’s worse.
What win rate is actually good for
I wouldn’t delete it. Win rate is a decent diagnostic, provided you read both ends of it:
- Very high (say, well above a third): you’re probably testing things you already knew, or your analysis is too generous. Audit for peeking and slicing before you celebrate.
- Very low, for a long time: your ideas aren’t grounded in any research, or your tests are too small to detect anything. Look at power and at where the ideas come from.
- Somewhere in the middle, at high coverage: that’s what the published base rates suggest a healthy, honest programme looks like.
Win rate is a smoke alarm. The trouble starts when a team treats it as the thermostat.
The scorecard I’d put in front of a board
If volume is the health indicator and win rate is a diagnostic, what should a programme actually report? This is the scorecard I use, in priority order:
- Coverage. Of the changes shipped this quarter where a reasonable person could have argued either way, what share went through a controlled experiment? Leave out bug fixes, legal requirements and the obviously broken. This is the headline number, and most teams have never calculated it.
- Decisions made. Tests concluded with a decision per month: ship, kill or iterate. This is volume, counted in the unit that can’t be padded. A test nobody acted on doesn’t count.
- Harm caught. Losing variants stopped before full rollout, with an estimate of what they’d have cost. This is the number that makes losing tests visible as the value they are.
- Measured cumulative impact. Not the sum of every winning test’s headline lift, which the winner’s curse guarantees is too high, but a long-running holdback: a slice of traffic that never gets the winners, compared against everyone else. It’s the only honest answer to “is all this testing actually making money?”
- Win rate, as a diagnostic on the table, never a target. Flag it if it’s suspiciously high.
A programme that improves on the first four is healthy almost by definition, and it will be running a lot of experiments. Not because anyone set a velocity target, but because you can’t get coverage any other way.
What changes when tests are cheap
Most of the case against volume was really a case against expensive volume. If every test needs engineering time, a QA cycle, and a client-side tool that slows the page and flickers the control, then high volume really does mean corners being cut somewhere.
Take the cost down and the argument falls apart. When a variant can be built, targeted and split at the edge in an afternoon, with no flicker, no page-speed penalty and a clean randomisation unit, testing stops being a scarce resource to ration and becomes the default way changes ship. That’s the point where coverage goes up without anyone working harder, and it’s the reason I built the tooling I did.
How we can help
The engagement usually has three stages, and the first one is worth having on its own.
First, find out whether you have this problem. We take last quarter’s shipped changes — releases, CMS edits, merchandising changes, redesigns — and set them against what was tested. That gives you a coverage number, often for the first time. Plenty of teams find their coverage is fine and what they actually have is a power problem or an analysis problem, and it’s better to know that before you buy anything. Some find they’re testing 10% of what they ship, and that is a very clarifying afternoon.
Then make testing the cheap path. Put the test layer at the edge, so a variant is a small, reviewable change rather than a sprint, and set up the defaults that stop volume turning into noise: one pre-declared primary metric, power checks before launch, and a decision rule agreed in advance.
Then measure whether it paid. Stand up a holdback so the programme’s cumulative effect is measured rather than added up from headline lifts. That’s the number that ends the volume-versus-quality argument, because it answers the question both sides were really asking.
Who this isn’t for
- Low-traffic sites. If you can power a handful of tests a year, coverage isn’t your lever. Research quality and hit rate are. Read the low-traffic guide instead.
- Teams that can’t ship quickly. If a change takes a quarter to get through release, cheaper tests won’t help. The bottleneck is upstream.
- Anyone who needs a win rate for the board. If the goal is a number that looks good in a slide, this approach will make that number worse before it makes the business better. That’s the point of it, but it’s a hard sell if the slide is what matters.
The point
The case against experiment volume is aimed at a real target and lands on the wrong one. Gamed volume — tests split to hit a quota, underpowered tests, tests nobody acts on — is worthless. But the answer to a badly counted metric is to count the right thing, not to stop counting.
Counted in decisions, volume is the best proxy for coverage there is, and coverage is the question that matters: of everything you changed this year, how much do you actually know the effect of? At 15% coverage, your tested wins are a small, well-measured island in a sea of untested changes, and a third of those, if the published base rates hold, are quietly making things worse. Some of them are overwriting the wins you already proved.
Experiments are research, and research is only worth doing on things you don’t know. That means testing the uncertain ideas, the bold ones and the unfashionable backlog items, and accepting that many of them will lose. A win-rate target punishes exactly that, and rewards the peeking, slicing and safe bets that make a programme look better while it gets worse.
So count your changes, count your tests, and divide one by the other. That number will tell you more about your programme than its win rate ever has.
Coverage is king.
If your coverage number turns out to be low, the fix is usually making tests cheap enough that testing becomes the default way changes ship, rather than a scarce slot someone has to argue for. That’s what Conversion Workers does: edge-delivered experiments with no flicker and no page-speed penalty, built in an afternoon rather than a sprint. If you’d rather start by finding out what your coverage actually is, get in touch. The audit is useful on its own, whatever it says.