Plenty of the decisions that matter most cannot be A/B tested.
You cannot randomise your SEO. You cannot show half your customers a different brand campaign on television. You cannot easily split a price change, and where you legally can, you often should not. And the release that shipped last Tuesday without a control group has already happened — the question “did it work?” still needs answering.
Randomisation is the gold standard precisely because it balances confounders you never thought of. Without it, you are estimating a counterfactual: what would have happened anyway. Every method below is a different strategy for constructing that counterfactual, and each buys its credibility with an assumption you should state out loud.
This expands the causal inference section of the cheat sheet.
The framework: potential outcomes
Every unit has two potential outcomes — one under treatment, one under control. Only one is ever observed. The unobserved one is the counterfactual, and causal inference is entirely the business of estimating it.
A confounder is anything that drives both the treatment and the outcome. A promotion running during your release. Seasonality during a redesign. The fact that the customers who adopted a feature were your most engaged ones already.
Randomisation balances confounders you measured and ones you did not. Every non-randomised method balances only what you measured and thought to include. That gap is the entire risk, and it is why these methods should be reported with more humility than an experiment.
Difference-in-differences
The workhorse. Compare the before-to-after change in a treated group against the same change in an untreated one:
τ = (Y_treated,post − Y_treated,pre) − (Y_control,post − Y_control,pre)Subtracting the control group’s change removes anything that affected both groups over the same period — seasonality, market conditions, a platform algorithm update.
The assumption is parallel pre-trends. The two groups must have been moving in parallel before treatment. If they were diverging already, DiD attributes that pre-existing divergence to your treatment.
Always plot the pre-period. Not a table — a chart, with several periods before treatment, so you and everyone reading can see whether the lines were parallel. If they were not, either find a better control group or use a method that does not require it.
Typical digital uses: a feature rolled out to one market first, a change applied to one product category, a policy applied to one customer tier.
Synthetic control
When no single control group looks like your treated unit, build one.
Synthetic control constructs a weighted combination of untreated units that reproduces the treated unit’s pre-treatment behaviour as closely as possible, then extrapolates that composite forward as the counterfactual. The gap between actual and synthetic after treatment is your estimated effect.
This is the strongest option for single-unit interventions — one market, one site, one brand campaign in one country. It is how you evaluate “we launched in Germany” or “we ran TV in the North West”.
What to check: the pre-period fit must be genuinely good (a poor pre-fit invalidates the whole thing), and the donor pool must contain units that were not themselves affected by the treatment. Placebo tests — running the same method on untreated units and confirming you find no effect — are the standard credibility check.
Regression discontinuity
Where a threshold determines treatment, compare units just either side of it.
Digital analytics is full of thresholds: free shipping over £50, loyalty tiers at a spend level, a ranking cutoff at position 10, a discount that applies at three items. Someone spending £49.90 and someone spending £50.10 are essentially identical people, but one gets treated and the other does not.
The estimate is local to the boundary — it tells you the effect for people near the threshold, not for everyone. That is a genuine limitation, but near the threshold the credibility is very high, close to experimental.
Watch for manipulation: if people can precisely control which side they land on (adding a cheap item to clear the free-shipping bar), the units either side are no longer comparable, and the design breaks.
Instrumental variables
An instrument is a variable that affects whether a unit gets treated, but has no path to the outcome except through the treatment.
It is the right tool under non-compliance — where you can encourage treatment but not enforce it. You emailed everyone about a new feature; some adopted it. Comparing adopters to non-adopters is hopelessly confounded, because adopters are different people. But the email itself was randomly assigned, and it can serve as an instrument for adoption.
Valid instruments are genuinely rare, and the exclusion restriction — no other path to the outcome — is untestable. Treat IV as a specialist tool and be sceptical of instruments that arrive too conveniently.
Propensity score matching
Model the probability of being treated from observed covariates, then compare treated units to untreated ones with similar propensity scores. Effectively, compare like with like.
Its limitation is fundamental and frequently glossed over: it balances only what you observed. If the real reason some users adopted a feature is enthusiasm you never measured, matching on tenure, plan and device does not fix anything. It makes the comparison look rigorous without making it causal.
Useful as a sanity check or a first pass. Dangerous as the sole basis for a spending decision.
Geo-lift testing
The practical bridge between experiments and observational work, and the standard for measuring media incrementality.
Split regions into treated and untreated — randomly where you have enough of them, matched where you do not — run the campaign in one set, and compare. Because assignment is under your control, this is an experiment; it just has a small number of large units.
Practical constraints: you have far fewer geos than users, so power is limited and only reasonably large effects are detectable. Spillover matters too — media does not respect regional boundaries, and neither do customers near them. Synthetic control is often the analysis method of choice for reading the result.
Choosing a method
| Situation | Method | Key assumption |
|---|---|---|
| Phased rollout, one market first | Difference-in-differences | Parallel pre-trends |
| Single market or site intervention | Synthetic control | Good pre-period fit |
| Treatment set by a threshold | Regression discontinuity | No manipulation at the cutoff |
| Encouraged but not enforced adoption | Instrumental variables | Exclusion restriction |
| Self-selected feature adoption | Propensity matching | No unobserved confounding |
| Media and brand spend | Geo-lift | Limited spillover |
Reporting these honestly
Observational causal estimates deserve more caveats than experimental ones, and the caveats belong in the summary rather than an appendix.
State the counterfactual you constructed and why it is credible. Show the pre-period. Name the assumption that would have to fail for your estimate to be wrong, and say what would make you doubt it. Give a range rather than a point estimate wherever you can, and run the obvious placebo test.
And when the observational result disagrees with an experiment, believe the experiment. These methods are for when randomisation is impossible — not for when it is merely inconvenient.
Where this connects
- Where randomisation is possible, do that instead: experiment design
- Why attributed conversions are not incremental ones: web measurement and attribution
- The statistical machinery underneath these estimates: hypothesis testing
- One-line definitions: the cheat sheet