Sooner or later every experimentation programme of any size gets the same question from finance: what does all this testing return? The usual answer is a list of winning tests with their lifts added up. Finance has learned not to believe it, and finance is right.
The return on a programme is a fraction. On top is the value of the decisions it produces: the wins that were kept and the harm that was caught before it shipped. Underneath is what those decisions cost. In a large organisation that cost is mostly people and waiting. A specialist team, engineering time for every variant, a platform licence, and weeks of calendar time in which an idea sits in a prioritisation forum, then a build queue, then an analysis queue, then a readout meeting.
The six items in this article, placed on the half of the fraction they act on. AI moves the bottom half first.
Most of what is being sold as “AI for experimentation” this year works on neither half. It generates test ideas, which were never the constraint, or it moves traffic around automatically, which helps far less often than the sales deck suggests.
I have run more than 6,000 experiments over twenty-odd years, and the pattern I see in corporate programmes is consistent. AI’s first and largest effect is on the bottom of the fraction. It makes a decision cheaper to reach. Whether it improves the top depends almost entirely on what the organisation has written down for it to read.
A note on evidence before the six. The figures in this article are either published base rates from programmes such as Microsoft’s and Booking.com’s, or arithmetic you can check with a calculator. I am not going to quote an uplift from an AI-assisted programme, because I have not seen one measured in a way I would defend. That gap gets its own section.
Pick one, or read them in order:
The three that make a decision cheaper
A programme’s return is capped by how much of what the business ships goes through a test. I have argued at length that coverage is the number that matters, and coverage is a cost problem. When a test needs a sprint and an analyst, most changes ship without one. Ronny Kohavi and colleagues reported that at Microsoft roughly a third of ideas improved the metric they were aimed at, a third did nothing and a third made it worse. Untested, that last third goes live and stays there.
Time spent waiting on a person. The numbers are the items in this article that shorten it. The run. Its length is set by traffic, and nothing here makes it shorter.
These three take cost out. Each is set out the same way: the challenge, the approach and the value.
1. Stop launching tests that cannot conclude
Challenge
The most expensive test in any programme is the one that runs for six weeks and ends in “inconclusive”. It used the traffic, the engineering time, the analyst and a slot in the roadmap, and returned nothing anyone can act on.
Most of those tests were dead before they launched. The page never had the traffic, the metric was too noisy, or another team’s test was running on the same step of the funnel. Everybody knows a design review would catch this. In a programme with forty squads and four specialists, the review is the first thing to be skipped.
Approach
Make the pre-flight something an agent does on every proposal, because unlike a specialist it is never too busy.
- Power. Illustrative arithmetic: at a 2% baseline conversion rate, detecting a 10% relative lift at 80% power needs roughly 80,000 visitors per arm. If the page gets 15,000 a week, that is a ten-week test, and somebody should decide whether it is worth ten weeks before it starts.
- Collisions. What else is live on the same pages, the same audience and the same primary metric.
- The decision. One primary metric, the guardrails and a stopping rule, written down before launch.
- History. Whether this has been tried before, which is the fifth item in the “Value per Decision” tab.
Arithmetic, not data from a test. Halve the lift you want to detect and you need roughly four times the traffic, which is why the check belongs before launch.
Value
Tests that cannot conclude are stopped at the proposal stage, where they cost a few minutes. The ones that go ahead start with their decision rule already agreed, which shortens the argument at the other end.
The specialist team stops being a gate that every test queues for and becomes the author of the checks. That is a better use of four people.
2. Build the variant without a sprint
Challenge
In most corporate programmes the binding constraint is engineering. A variant competes with the roadmap for a place in the sprint, and usually loses. So tests are rationed, the rationing is called prioritisation, and the programme ends up testing a small, carefully chosen share of what ships.
Rationing has a cost that never appears on a dashboard. Kohavi and Stefan Thomke described a Bing change to how ad headlines were displayed that sat in the backlog for more than six months because it was judged low priority. When someone finally ran it, it lifted revenue by 12%, worth more than $100 million a year in the US alone.
Approach
Coding agents are now good enough to build most variants: a layout change, a reordered form, a different default, the same component behind a feature flag. Give the agent the design system, the flagging conventions and the experiment standards, and have it produce the variant as a pull request.
The rails matter more than the generation. A human still reviews the code. The variant ships dark behind a flag. Regulated journeys and legally fixed wording are listed as out of bounds. And the flag is removed when the test ends, a job the agent can also be given and one that almost nobody does.
Value
The cost of a variant falls from a place in the sprint to a code review. That changes which ideas get tested. The backlog item nobody would fight for becomes cheap enough to run anyway, and that is the category the Bing result came from.
It also makes a more ambitious goal realistic: testing as the default way a change ships, not a special project that has to be argued for.
3. Clear the analysis queue
Challenge
A test that finished on Friday and is read out three weeks later has cost three weeks of the winning variant not being live, or three weeks of a losing one still taking half the traffic. In most programmes the analysts are the queue, and most of what sits in it is routine: the same readout, to the same template, for a test with a pre-declared metric.
Approach
Have an agent produce the standard readout the morning a test reaches its planned end. It works against the metric definitions the organisation has already agreed and follows the house rules described in the fourth item. Sample ratio check first, then the primary metric with its interval, then the guardrails, then a plain statement of what the pre-agreed decision rule says.
Analysts review by exception. Anything with a sample ratio mismatch, a guardrail breach or a result that contradicts an earlier test goes to a person. So does anything where the agent’s output fails its own checks.
Value
Decision latency drops, and analyst time moves to the tests that need judgement.
The caveat is the important part. If every AI readout needs an analyst to check it, the workload has not gone away. It has turned into a review queue. The saving only exists once the outputs are being evaluated systematically, which is the argument of evals are the bottleneck.
The three that make a decision worth more
Cheaper decisions raise the return only if they are the right decisions. These three work on the top of the fraction: whether the result can be believed, whether the organisation remembers it, and whether the total can be defended in front of finance.
Same shape: challenge, approach, value.
4. Stop shipping false winners
Challenge
An agent asked “did it win?” is a tireless producer of wins. It will look every day, try a second metric, and slice by segment until something clears the bar. People do this too, but they get tired.
The arithmetic is unkind. Cut a flat test into twenty segments at the usual 5% threshold and the chance that at least one shows a significant “effect” is 64%. That is the multiple comparisons problem. In a programme where teams are judged on win rate, it turns from a statistical footnote into the way targets get hit.
Arithmetic, not data from a test: one minus 0.95 to the power of the number of comparisons, treating them as independent. An agent can run twenty cuts in the time a person runs one.
Approach
Write the house rules down once and have the agent apply them to every test. The primary metric is the one declared before launch. The stopping rule is followed. A segment is only a result if it was named in advance, and anything found afterwards is a hypothesis for the next test. Guardrails are checked before any win is accepted.
Then turn the same rules on the past. An agent can audit last year’s reported wins against them in an afternoon. That is the fastest way I know to find out how much of a programme’s claimed value would survive.
Value
Fewer wins, and wins that show up in the numbers. A false winner costs twice: the engineering to ship and maintain it, and the credibility lost when the forecast lift never arrives.
Applied consistently, written rules do something a centre of excellence struggles to do by persuasion. Every squad’s tests are read the same way, including the ones run by the team with the most senior sponsor.
5. Stop re-running what you already know
Challenge
Large programmes forget. Results live in slide decks, wiki pages and the heads of people who have since moved teams. Two years later a new product manager proposes the test that lost in 2024, and nobody in the room knows.
It is waste on both sides of the fraction. Traffic and build time go on a question already answered, and the learning that should have shaped the next ten tests was never available to the people designing them.
Approach
Put the history where a machine can read it: one record per test, with the hypothesis, the audience, the primary metric, the result with its interval, the decision and the takeaway. An agent can do most of the backfill from old decks and tickets, with a person confirming each record. The same agent then checks every new proposal against the log.
This is one of a set of documents I have described in context engineering for experimentation, and it is the one that compounds.
Value
Duplicate tests stop at the proposal stage. Proposals get sharper, because “we tried this on mobile and it lost, here is what was different” is a better starting point than a blank page.
It also opens up the analysis no large programme gets round to: which kinds of change tend to work here, on which parts of the journey, for which customers. Every organisation running the same model as you has access to the same generic best practice. Nobody else has your results.
6. Report the programme in money
Challenge
Back to finance. The programme reports its winners and adds up their lifts. The total is too high, for a reason that has nothing to do with dishonesty. Effects that only just clear the bar are systematically overestimated, which is the winner’s curse. The same sum leaves out everything that shipped untested, and it values a conversion lift on a low-margin product the same as one on a high-margin product.
An AI optimising a conversion metric with no knowledge of margin makes the last problem worse. The quickest route to more orders is usually cheaper orders.
Approach
Three things, in rising order of effort.
- Give the agent the economics. Margin by product line and customer type, and the guardrail metrics a win is not allowed to breach. A win that moves the mix towards low-margin sales is reported as what it is.
- Count what the programme prevented. Every losing variant stopped before rollout is harm that did not ship. It is the most undervalued number a programme has, and an agent can put an estimate on it from readouts that already exist.
- Run a holdback. A slice of traffic that never gets the winners, compared with everyone else over months. It is the only honest answer to the question finance is asking.
The second of those is a picture I have been drawing for years, usually for someone commiserating over a losing test:
A losing test is a change that would otherwise have shipped. The programme's value is the gain it found plus the loss it prevented, and most reports count only the first.
Value
The programme can state its return in a form finance will accept: measured, net of the winner’s curse, in margin and not conversion rate. Why that distinction matters so much is the subject of the revenue metrics dirty secret.
The number is usually smaller than the sum of the wins, and it is far easier to defend at budget time.
Where it fails
The first two tabs describe AI in a programme when it is set up well. These are the ways it goes wrong. I would read this tab before agreeing a budget for any of the above.
The review queue
The commonest outcome of an AI analytics pilot is that every output gets checked by an analyst before anyone trusts it. That is understandable and it removes the saving. The pilot reports that the model was accurate, the analysts report that they are busier, and both are right.
The way out is not better prompting. It is deciding what a correct readout is, testing the agent against a set of past experiments where the answer is known, and reviewing by exception once it passes. Without that step, the third item in the “Cost per Decision” tab is a cost.
The win factory
Give an agent a target and no rules and it will hit the target. In a programme measured on win rate, AI-assisted analysis will raise the win rate through peeking, slicing and metric switching, done faster and more thoroughly than a person would manage. The dashboard improves. The revenue line does not.
Nobody has to intend it. It follows from the incentive, and it is why the rules in the fourth item have to exist before the agent gets near a results page.
The context that went stale
An agent follows written definitions precisely. That is the point of writing them, and it is also the hazard. A metric definition that was right last year and is wrong now will be applied to every test, consistently, and the output will look as tidy as ever. A human analyst would have heard about the change in a corridor.
The documents need an owner and a review date, the way code does. An out-of-date definition is worse than none, because it is wrong with complete confidence.
More tests, more collisions
Take the cost of a test down and the number running at once goes up. Tests start to overlap on the same pages and audiences. Shared metrics get noisier. The platform team that could support twenty concurrent tests is now being asked to support eighty.
None of this is a reason to keep tests expensive. It does mean that assignment, exposure logging and collision checks have to be sound before the volume arrives. An agent cannot fix that by writing more code.
The same ideas as everyone else
Ask a model for test ideas with no context and you get the contents of every optimisation listicle it was trained on. Your competitors, using the same model, get the same list. If AI-generated ideation is the centrepiece of the plan, the plan amounts to running the industry’s average backlog a little faster.
The AI your vendor is selling
The most heavily marketed “AI” in experimentation is not a language model. It is the multi-armed bandit: traffic shifted automatically towards whichever variant is ahead. For outcomes that arrive straight away, such as a click on a headline, it works well.
For outcomes that take days it is a different matter. Once the split moves, the variants no longer hold the same mix of new and returning visitors. I worked through the arithmetic in multi-armed bandits and return-user bias: two identical variants, both truly converting at 5%, read 4.43% and 3.77% after the split moved, which is 15% apart with no difference between them. Measure how long your customers take to convert before you pay for the feature.
The thing nobody can yet claim
I can tell you with confidence that AI makes a test cheaper to design, build and read. I cannot tell you, with a measurement I would defend, how much more a programme earns as a result. I do not have a holdback result from an AI-assisted programme to quote, and I have not seen anybody else publish one. Vendor case studies are sums of headline lifts, which the sixth item explains the problem with.
So treat every claim about AI and experimentation uplift, including any you are being sold this quarter, as unmeasured until someone shows you the holdback. Cheaper is demonstrable. More profitable is still an assumption.
The order to do this in
The six are a sequence, not a menu. The parts that look least like AI projects come first, because they decide whether the rest make money or only make the programme busier.
First, count your coverage. Of the changes that shipped last quarter where a reasonable person could have argued either way, how many went through a test? Most organisations have never worked it out. It tells you how much there is to gain from cheaper tests, and whether cost is your constraint at all.
Second, find where the calendar time goes. Take the last twenty tests and measure the days between idea, launch, end of test and decision. If the longest gap is before launch, start with the first two items. If it is after the test ends, start with the third.
Third, write the rules and the economics down. Primary metric before launch, the stopping rule, how segments are treated, which guardrails block a win, and what a sale is worth by product line. A few pages. The agent reads them on every test, and so does every new hire.
Fourth, build the experiment log. Backfill what you can, and make a record the price of closing any new test.
Fifth, automate the pre-flight and the readout against those rules, with an evaluation set of past tests, and review by exception once it passes.
Sixth, bring build cost down with coding agents behind flags and a human code review.
Throughout, run the holdback. It takes months to say anything, so it should start first and report last.
Who this is not for
- Programmes that cannot power a test. If the traffic supports six tests a year, cost per test is not the constraint. The low-traffic playbook is a better use of the afternoon.
- Organisations where release is the bottleneck. If a change takes a quarter to ship, a variant built in an afternoon sits in the same queue.
- Organisations where results do not change decisions. If the senior sponsor ships the redesign whatever the test says, a faster readout only brings the disappointment forward. That is a leadership problem and AI does not touch it.
- Anyone who needs a higher win rate for a slide. Stricter readouts will lower it before they improve anything else.
How Conversion Works helps
Everything above can be built in-house, and the article is written so that you can. The hard parts are the order, the rails and finding out afterwards whether it paid.
We start by finding out whether you have the problem. A coverage count for last quarter, a timeline of where the days go on your last twenty tests, and an audit of last year’s reported wins against a written set of rules. It is the smallest piece of work we do, and it is useful on its own. It often concludes that the constraint is traffic, release or decision-making, not the cost of a test, in which case an AI programme is not the thing to buy.
Then we set it up with the rails attached. The written rules and economics, the experiment log, the pre-flight checks, a readout pipeline with an evaluation set behind it, and the approval steps that keep a person between an agent and a customer. It is handed over as something your team runs and owns.
Then we measure whether it paid. A holdback, so the programme’s effect is measured and not added up. It is the step that gets skipped, and the only one that answers the question in the title.
Behind that is twenty-plus years in experimentation and analytics, more than 6,000 experiments and two production products built for this work.
The point
The return on an experimentation programme is the value of its decisions divided by their cost. In 2026 and into 2027, what AI reliably changes is the cost. It can check a proposal before it wastes six weeks of traffic, build the variant without a sprint, and produce the readout the morning the test ends.
That matters because cost is what holds coverage down, and coverage is what a programme is for. But a cheaper decision is only worth having if it is right, remembered and counted in money. Those depend on things an agent cannot invent: what counts as a result, what has been tried before, and what a sale is worth. Give it those in writing and it applies them more consistently than a stretched specialist team can. Leave them out and it will produce winners faster than the organisation can ship them, and very few will reach the revenue line.
So the work is less about the model than about a handful of documents and one long-running holdback. Count your coverage, write the rules, build the log, make testing the cheap way to ship, and measure the whole thing against customers who never got the winners.
Everyone has the same models. The programmes that get a return from them will be the ones that wrote down what they know.
Two things have to be true underneath all of this. The data the agent reads has to mean what the definitions say it means, which is what Conform protects by enforcing event schemas at the edge. And a test has to be cheap and clean to run, with the variant decided before the page renders and assignment that stays with the visitor, which is what Conversion Workers does. If you would rather begin by finding out what your coverage is and where your calendar time goes, get in touch. Both numbers are worth having whatever you decide to do next.