Almost every article about A/B testing is written for a site with enough traffic that the maths stays out of the way. Most sites are not that site. A £2m e-commerce business, a B2B site with 400 demo requests a year, a subscription product with 3,000 monthly visitors — these all want the same thing from experimentation, and almost none of the standard advice survives contact with their numbers.

The usual response is one of two failures. Either the team decides testing “isn’t for us” and ships on opinion forever, or — far more common and far more damaging — they run underpowered tests anyway, read the results as if they were valid, and accumulate a portfolio of imaginary wins.

There is a third option, and it has two halves. Change what you test and how you group it, so you need less traffic. Change how you analyse and decide, so you extract more from the traffic you have and stay honest about what you still don’t know.

First, the arithmetic that causes the problem

For a binary metric like conversion rate, the sample you need per arm at the usual α = 0.05 two-sided and 80% power is roughly:

n ≈ 16 · p(1−p) / Δ²      per arm

Where p is your baseline rate and Δ is the absolute difference you want to detect. The important term is Δ². Detecting an effect half the size costs four times the traffic. It is a quadratic wall, and no amount of tooling gets you over it.

Here is what that means at a 2.5% baseline conversion rate:

Relative lift you want to detectAbsolute ΔSessions needed (both arms)Weeks at 2,000 sessions/wk
5%0.13pp~499,000250
10%0.25pp~125,00062
20%0.50pp~31,00016
30%0.75pp~14,0007
50%1.25pp~5,0002.5

Read that table properly and the strategy writes itself. At 2,000 sessions a week you are not in the business of detecting 5% lifts. Ever. You are in the business of detecting 25%+ lifts, roughly four to six times a year. Everything below is about making that a viable programme rather than a consolation prize.

One thing to be clear about first: low traffic does not make your tests less accurate, it makes them less sensitive. An underpowered test is not a slightly blurry version of the truth. It is a test that will usually return “no significant difference” when a real effect exists, and will badly exaggerate the effect on the rare occasions it does return a winner. More on that in part two.

Part one: what to change about the programme

Test bigger things

Δ is squared in the denominator and it is the only term you meaningfully control. So control it.

Small changes produce small effects. A button colour, a microcopy tweak, a repositioned trust badge — these are the bread and butter of high-traffic CRO precisely because those programmes can afford to detect a 2% lift. On a low-traffic site, a button-colour test is not a cheap experiment. It is an expensive one that will conclude nothing after eight weeks and cost you the only test slot you had that quarter.

What actually produces detectable effects:

  • Whole-page redesigns rather than element tweaks. Yes, you lose the ability to attribute which element did it. You were never going to get that attribution at this traffic level anyway, so stop paying for it.
  • Structural changes to the flow. Removing a step from checkout. Collapsing a two-page form into one. Moving required account creation to after purchase.
  • Offer, price and proposition changes. Free delivery thresholds, guarantees, finance options, bundling. These move behaviour by tens of percent, not fractions.
  • Adding something that isn’t there at all. A missing product-page element — sizing guide, stock indicator, delivery date, review block — usually beats optimising an element that already exists.

The instinctive objection is that big changes are riskier. That is true, and it is also the point. A big change has a wide distribution of outcomes; a small change has a narrow one centred near zero. When you only get a handful of shots, you want variance in the outcome, not precision around nothing.

Group pages together and test the template

The most useful structural move on a low-traffic site is to stop thinking about pages and start thinking about templates and cohorts of pages.

You may have 800 product pages, none of which individually carries enough traffic to test. Collectively they carry all of it. So change the product-page template and randomise users across the whole set. Same for category pages, article layouts, location pages, or any programmatically generated set.

Practical ways to pool:

  • Template-level tests. One change, applied to every page using that template, users randomised once at the site level.
  • Section-level tests. Everything under /blog/, one product category, one funnel stage.
  • Site-wide tests. Header, footer, navigation, delivery-promise messaging, cart drawer. These reach almost everyone, which maximises the effective sample.
  • Multiple properties. If you run several sites, the same change tested on each gives you several independent estimates you can combine (see meta-analysis below). Don’t just run it on the biggest and assume the rest follow.

Two caveats worth stating honestly. First, you are now estimating an average effect across a heterogeneous set of pages — the change might help cheap products and hurt expensive ones, and the average will hide it. Pre-register one or two segments you’ll look at, and accept that you can’t go mining afterwards. Second, randomise the user, not the page. Randomising pages into arms sounds appealing but makes the page the unit of analysis, and 800 pages is a far smaller sample than it feels like.

Trigger the analysis to the people who could possibly be affected

This is the largest free sensitivity gain available to most teams, and it costs nothing but discipline.

If you change something in the delivery information section and analyse every visitor to the site, you have diluted a real effect across a large majority who never scrolled to it. If 20% of visitors see the change and it lifts their conversion by 25%, the diluted site-wide effect is 5% — which you cannot detect, when the real effect was comfortably detectable.

Fire an exposure event at the point of eligibility — the point where a user would have seen the change, defined identically in both arms — and analyse only triggered users. Getting this right routinely turns an impossible test into a feasible one.

The trap: instrument the trigger before the branch. If your variant fires an event when a new component renders and the control has no such component, you are comparing triggered users in one arm against everyone in the other, and you have built a broken test rather than a sensitive one.

Run one test at a time

High-traffic programmes run tests in parallel and treat interaction effects as noise. You can’t. Every concurrent test either splits your traffic or contaminates the other’s arms.

Run a serial queue — which makes prioritisation the highest-leverage activity in the whole programme, because you are choosing perhaps six ideas a year rather than sixty. Prioritise on traffic exposure × plausible effect size × reversibility, and be ruthless about the first term. An idea for a page carrying 4% of your sessions is not an idea, it’s a distraction.

Ship the obvious things without testing them

Experiment capacity is your scarcest resource. Don’t spend it proving things you already know.

Ship without a test: bug fixes, broken layouts on real devices, page-speed improvements, accessibility fixes, anything legally required, anything where the current state is plainly defective. Testing a fix for something broken isn’t rigour — it’s spending six weeks of your annual measurement budget to confirm that broken is worse than not broken.

Reserve the test slots for decisions that are genuinely uncertain: where a reasonable person could argue either way, and where being wrong costs real money.

Use non-experimental evidence to raise your hit rate

When you only get six tests a year, win rate matters more than measurement precision. A programme running 60 tests at a 15% win rate gets nine wins. A programme running six tests needs a much better strike rate to justify itself, and that comes from research, not statistics.

So spend the time you’re not spending on analysis on:

  • Session replay and funnel analysis, to find where people actually fail rather than where you assume they do.
  • Moderated usability testing. Five sessions will surface most severe usability problems. Not a substitute for measurement, but a very good filter on what deserves measuring.
  • Customer service and sales transcripts. The objections costing you money are already written down somewhere.
  • On-site surveys — exit intent, post-purchase, “what nearly stopped you?” These work fine at low traffic because a few hundred responses is plenty.
  • Painted-door tests. Advertise a feature that doesn’t exist yet and measure clicks. A click is a far higher-frequency event than a purchase, so demand questions are answerable at traffic levels where conversion questions are not.

None of this is an A/B test. All of it makes your six A/B tests more likely to be about something that matters.

Run whole weeks, but know the ceiling

Run complete weekly cycles, minimum two. Weekday and weekend traffic differ in intent, device and basket, and stopping mid-week bakes in whatever mix you happened to catch.

But running longer is not an infinite lever. Past roughly six weeks you start accumulating real problems: cookie churn and device switching mean the same human gets reassigned across arms, seasonality drifts underneath you, the business changes things you didn’t control for, and the returning-visitor population accumulates novelty and primacy effects. Six weeks is a practical ceiling for most sites. If your design needs twelve, the design is wrong — go back and make the change bigger, the trigger tighter, or the metric more sensitive.

Move upstream — carefully

A metric further up the funnel has a higher base rate, which means more events and more power. Add-to-cart happens perhaps ten times more often than purchase; email signup more often still.

This is a legitimate technique and also the most abused one in low-traffic CRO. Moving to a proxy only works if you have established that the proxy actually predicts the thing you care about, in your business, with your traffic. Add-to-cart rate is easy to lift by making the button more prominent — and equally easy to lift without moving a single extra order.

Before adopting a proxy, check the historical user-level relationship between the proxy and revenue. If it’s weak, you’re not measuring a leading indicator, you’re measuring a distraction. And whichever proxy you choose, keep the real metric as a directional guardrail: you want to know if orders fell while add-to-carts rose.

Consider a holdout instead of a test

If the constraint is genuinely binding — you want to make twenty improvements this year and can only measure four — invert the model. Ship the work, and hold back a randomly assigned slice of users from all of it.

A 10% holdout maintained for a quarter answers the question “is the sum of everything we did worth anything?” It won’t tell you which change did what. But an honest answer to the aggregate question beats twenty individually unmeasurable claims, and it’s the standard approach in personalisation and recommendation systems for exactly this reason: aggregate effects are larger than individual ones, so they need less traffic to detect.

Part two: what to change about the statistics

Everything above buys sensitivity by changing the experiment. This half is about extracting more from the data you do collect — and making defensible decisions when the data can’t settle the question.

Variance reduction: the closest thing to free traffic

Variance sits in the numerator of the sample-size formula. Reduce it and you reduce the required sample proportionally, without touching a single extra visitor.

CUPED (Controlled-experiment Using Pre-Experiment Data) is the main technique. The intuition: some users are simply heavier spenders than others, and that variation has nothing to do with your treatment. If you know how each user behaved in a pre-experiment period, subtract that predictable component and compare what’s left.

Y′ = Y − θ(X − X̄)     where θ = cov(Y,X) / var(X)

The variance reduction is 1 − ρ², where ρ is the correlation between pre-period and in-experiment behaviour. A correlation of 0.5 removes 25% of the variance — a 25% cut in required sample, free. On revenue metrics for sites with returning customers, correlations of 0.4–0.6 are common.

The obvious limitation on a low-traffic consumer site: many users are new and have no pre-period, and CUPED gives you nothing for them. Two things that still work:

  • Covariate adjustment more generally (ANCOVA, or regression with covariates). Use anything measured before exposure and correlated with the outcome: device, channel, landing page, day of week, geography, logged-in state. Never a post-exposure variable — that reintroduces exactly the bias randomisation removed.
  • Stratified randomisation. Balance the arms deliberately across device or channel rather than trusting randomness to do it at small n. At low sample sizes, chance imbalance is a real and underrated source of noise.

Fix the metric definition before you fix the analysis

Revenue per visitor is a terrible metric at low traffic because it is heavy-tailed. One £1,800 order in a 3,000-visitor arm can flip the result on its own. Its coefficient of variation is often 5–10× that of a binary conversion metric, which means many times the sample for the same sensitivity.

Cheap fixes, in order of preference:

  1. Decompose it. Test on conversion rate (bounded, low variance) and report AOV separately as context. Most changes act on one or the other, not both.
  2. Winsorise or cap. Cap order values at the 99th percentile — decided and written down before you look at the data. Post-hoc outlier rules are p-hacking with extra steps.
  3. Binarise. “Did this user spend more than £X?” throws away information, but throws away far more variance than information.

Also cut the junk before it reaches the analysis: bot and crawler traffic, internal IPs, staging referrals, and — if your business has them — trade or wholesale accounts whose order values sit an order of magnitude off the consumer distribution.

Understand what an underpowered “win” actually is

This is the most important statistical idea on this page, and the one most likely to save you money.

Suppose your test has 20% power for the effect that’s really there. Run it, and 80% of the time you get a null. But the 20% of the time you do cross significance, the estimate you’re looking at is not an unbiased view of the truth. It crossed the threshold precisely because noise pushed it up, so the observed effect is systematically inflated. At around 20% power, significant estimates run on average roughly two to three times the true effect — statisticians call this the exaggeration ratio, or Type M error. At very low power there is also a non-trivial chance the significant result has the wrong sign altogether.

The practical consequences:

  • Never treat the point estimate from an underpowered winner as a forecast. If it says +30%, budget for +10% and be pleasantly surprised.
  • Expect regression on replication. This is the mechanism behind “our A/B wins never show up in the P&L”. It isn’t the tooling. It’s the winner’s curse.
  • Discount underpowered results explicitly when they feed a business case. Shrink them (see hierarchical pooling below) rather than quoting them raw.

The corollary is more optimistic than it sounds. An underpowered test showing a large effect with a tight interval — because the effect really was large — is perfectly trustworthy. The problem is marginal significance, not low traffic as such. Which is precisely why part one told you to test bigger things.

Choose your error trade-off deliberately, and say so

α = 0.05 and 80% power are conventions inherited from a completely different context. They are not laws, and treating them as such at low traffic means never deciding anything.

The honest framing is decision-theoretic: what does each kind of mistake actually cost?

  • Type I error (shipping a change that does nothing): for a cheap, reversible front-end change — approximately zero. You shipped a variant that is no worse.
  • Type II error (failing to ship a real improvement): the full value of that improvement, forever, plus the opportunity cost of the test slot.

When the second cost dominates — which for reversible UX changes it usually does — running at α = 0.05 isn’t conservative, it’s mis-calibrated. Setting α = 0.10 or 0.20 for low-stakes reversible decisions is defensible and buys real power. What is not defensible is choosing α after seeing the result. Write it in the test plan, with the reason.

Similarly, a one-sided test is appropriate when the decision genuinely is one-sided (“ship if better, keep control otherwise”) and buys a little power for free. Just don’t quietly switch to one-sided when two-sided didn’t work out.

The bigger point: pre-register the MDE, and be willing to abandon the test before running it. If the honest calculation says “the smallest effect we could detect in six weeks is 40%”, ask whether the change plausibly delivers 40%. If not, don’t run it — ship on judgement and monitor, or do something else. A test you know in advance cannot answer the question is worse than no test, because it produces a number people will treat as an answer.

Report intervals, decide on intervals

Stop reporting “not significant”. It is the least informative sentence in analytics, and at low traffic it will be your most common result.

Report the confidence (or credible) interval and make the decision on its range:

  • “+18%, 95% CI [−4%, +42%]” — probably fine, plausibly good, unlikely to be badly harmful. On a reversible change, ship it and keep watching.
  • “+2%, 95% CI [−25%, +31%]” — you learned nothing. Say so plainly.
  • “−1%, 95% CI [−4%, +2%]” — a genuinely informative null. You’ve ruled out anything large in either direction, which is a real finding.

All three are “not significant”, and they license completely different decisions. The interval carries the information; the binary verdict throws it away.

Bayesian analysis for proportionate decisions

Bayesian methods do not manufacture information from nothing, and anyone selling them as a low-traffic cure is overselling. What they do is produce outputs that map onto the decision you actually have to make:

  • P(variant > control) — a directly interpretable probability, rather than a statement about hypothetical repeated sampling.
  • Expected loss — the average amount you give up if you ship the variant and you’re wrong. Set a threshold (“ship when expected loss < 0.3% of conversion rate”) and you have a coherent, pre-registered decision rule that behaves sensibly at any sample size.

That expected-loss framing is what makes this genuinely useful at low traffic. A frequentist null forces you into a default action by omission. Expected loss asks: given everything we know, how bad can this decision be? When the answer is “barely at all”, shipping the variant is rational even at 70% probability of being better.

Where Bayesian methods do add real information is through priors. If you’ve run twenty tests and the effects clustered between −5% and +10%, that is genuine, hard-won knowledge about your site. Encoding it as a weakly informative prior both stabilises small-sample estimates and automatically shrinks implausible ones. A flat prior treats “+300% conversion lift” as exactly as plausible as “+3%” before seeing data — which nobody actually believes.

One warning: a Bayesian analysis with a flat prior and a “ship when P(better) > 95%” rule, peeked at daily, has broadly the same false-positive behaviour as peeking at p-values. The framework doesn’t exempt you from the stopping-rule problem.

Sequential testing, used for the right purpose

Always-valid inference — group sequential designs, alpha spending, mSPRT — lets you look continuously without inflating error rates. It’s a real advance, and most decent platforms now offer a version of it.

At low traffic, be clear about what it buys you. It will rarely let you stop early with a winner, because you were never going to accumulate enough evidence early. What it does buy is the ability to abandon losers and hopeless flat tests early, returning your scarcest resource — a test slot — to the queue. That’s worth having when you only get six a year.

The trade is a modest loss of power relative to a fixed-horizon test at the same total sample. Given the discipline it enforces, it’s usually worth paying.

Pool across tests and pages: hierarchical models and meta-analysis

Individual low-traffic experiments are noisy. A set of related low-traffic experiments contains far more information than any one of them, and there are two ways to use it.

Hierarchical (partial pooling) models. Instead of estimating each variant or segment independently, estimate them jointly under a shared distribution. Effects get pulled towards the group mean by an amount that depends on how noisy each individual estimate is. This is the principled correction for the winner’s curse — noisy extremes shrink hard, well-measured estimates barely move. It’s especially useful on template tests where you want per-category or per-segment effects you have no hope of powering individually.

Meta-analysis across your own programme. After a year, pool your results by class of change: navigation, checkout, product page, trust and reassurance, pricing presentation. You’ll learn things no single test told you — which categories of change actually move your numbers, and what your realistic effect-size distribution looks like. That distribution then becomes the prior for everything you run next, and the input to your MDE decisions. Very few small teams do this. It compounds.

When you genuinely cannot randomise

Some changes can’t be A/B tested: pricing you’re only willing to show one way, site-wide infrastructure, brand changes, anything where showing different users different things is a legal or commercial problem.

The quasi-experimental toolkit applies:

  • Interrupted time series — model the pre-period trend and seasonality, then measure the deviation after the change. Needs a decent run of clean history and a sharp, known change date.
  • Difference-in-differences — compare changed pages against a comparable set of unchanged pages over the same window. Often available on low-traffic sites precisely because you have hundreds of pages: change one category, hold another as control, and the shared time trend cancels.
  • Synthetic control — build a weighted combination of untreated pages or regions that tracks the treated unit’s pre-period behaviour, then compare after.
  • Switchback / time-split designs — alternate the whole site between control and treatment on a schedule (day on, day off; alternating weeks). Suits changes that must apply to everyone at once. Beware: the effective sample is the number of time blocks, not visitors, and day-of-week effects will eat you if the schedule isn’t balanced.

These are weaker evidence than a randomised test and should be reported as such, with their assumptions stated explicitly. But “weaker than an A/B test” is a much better place to be than “based on a meeting”.

What not to do

The failure modes are worse at low traffic, not better, because there’s less signal for the discipline to protect:

  • Running until significant. With small samples the estimate wanders enormously, so “check daily, stop when green” will find you a winner nearly every time. Almost all of them are noise.
  • Segment mining. Twenty segments at α = 0.05 gives you roughly a coin-flip chance of at least one spurious “significant” segment in a test with no effect at all. And every segment in a low-traffic test is underpowered by construction, so every segment result is maximally exaggerated. Pre-register one or two, or report segments descriptively with no significance claims.
  • Multiple primary metrics. One metric decides the test. If two can, you’ve doubled your false-positive rate and given yourself room to pick whichever looks better afterwards.
  • Quietly dropping a bad week. If you exclude data, the rule must be pre-specified and applied identically to both arms.
  • Reporting “no significant difference” as “no difference”. At 20% power, absence of evidence is barely evidence of absence at all.

Putting it together: what’s viable at what traffic

Monthly conversionsRealistic approach
Under ~100Don’t A/B test. Research, usability, painted doors, upstream proxy metrics, quasi-experiments. Ship on judgement and monitor with control charts.
~100–500Two to four tests a year, site-wide or template-level, targeting 30%+ effects. Relaxed α, interval-based decisions, expected-loss framing.
~500–2,000Six to ten tests a year. Structural changes targeting 15–25%. CUPED and covariate adjustment worth building. Sequential rules to kill losers early.
~2,000–10,000A real programme. 10–20% MDEs, some element-level testing, hierarchical pooling across segments, meta-analysis of the back catalogue.

Moving through those tiers isn’t only about waiting for traffic to grow. It’s about running the programme now so that when the traffic arrives you already have the instrumentation, the triggering discipline, the pre-period covariates and the effect-size priors that make the next tier work.

The bottom line

Low traffic doesn’t remove your ability to make evidence-based decisions. It removes your ability to make cheap, precise, frequent ones. So you make fewer decisions, about bigger things, pooled across more surface area, with better research behind them and more honest uncertainty attached to them.

The failure isn’t “we can’t run A/B tests”. The failure is running them anyway, pretending they’re powered, and letting a portfolio of noise harden into a strategy.

Where this connects