Almost every statistical method used in product analytics assumes something about the shape of your data. Almost every metric in product analytics violates it.

That sounds like a crisis and mostly is not, because of one theorem. But you need to know which assumptions you are leaning on, and when they stop holding — because the point at which they break is exactly the point where a test starts producing confident nonsense.

This expands the descriptive statistics section of the cheat sheet.

Your metrics are not bell curves

Take revenue per session on any store with a range of price points. Most sessions produce nothing. A few produce £20. Occasionally someone buys a £900 item. The distribution is a spike at zero, a hump, and a long right tail.

Session duration: same shape. Items per order: same. Time to first purchase: same. Right-skew is the default in digital products, not the exception.

Two consequences follow immediately.

The mean and median tell different stories. On right-skewed data the mean sits well above the median and moves whenever a large value lands. If your AOV chart jumped last Tuesday, the first question is not “what did we change” — it is “how many orders drove that, and were they all one customer?”

Averages are unstable at low volume. The mean of a heavy-tailed distribution converges slowly. Two identical weeks can produce visibly different means purely by where the big orders happened to fall.

Variance is the number that costs you money

Standard deviation gets taught; variance is what you actually pay for. It determines how much traffic an experiment needs, and therefore how many experiments you can run in a year.

The standard error of a mean is:

SE = σ / √n

The square root is the expensive part. Halving your uncertainty requires four times the sample. This is why “just run it a bit longer” is rarely a small ask, and why variance reduction techniques like CUPED — covered in experiment design — are worth real engineering effort.

For a proportion such as conversion rate:

SE(p̂) = √( p(1−p) / n )

Note what that implies: variance is highest at p = 0.5 and falls as you approach 0 or 1. A 50% conversion rate is noisier per observation than a 2% one — but the relative precision at 2% is far worse, which is why low-converting funnels need so much traffic.

Comparing metric noisiness

The coefficient of variation, CV = σ/μ, makes metrics comparable:

MetricTypical CVImplication
Conversion rateunder 1cheapest to test
Items per order1–2moderate
Revenue per session4+expensive, needs volume or capping

This single table explains most “why did that test take six weeks” conversations. If you want to decide on revenue per visitor, budget for it.

The central limit theorem is your licence

Here is why skewed metrics do not immediately invalidate everything: the central limit theorem says that the distribution of the sample mean tends toward normal as n grows, whatever the shape of the underlying data.

Your revenue distribution is grotesque. The distribution of your average revenue across 40,000 sessions is close enough to normal for a t-test to behave.

The caveats matter though:

  • “As n grows” is doing work. With extreme skew you need more data before the approximation is good. A few hundred observations is not enough for revenue; tens of thousands usually is.
  • Extreme tails slow convergence. If a single observation can be 500 times the mean, the sample mean is still lumpy at surprisingly large n.
  • It says nothing about individual observations. You cannot use it to make claims about typical users.

When you are unsure whether you have enough n for the approximation, do not agonise — bootstrap it. Resample your data a few thousand times, look at the empirical distribution of the mean, and see whether it is symmetric. That is an afternoon’s work and settles the question.

Outliers: decide before you look

This is where good analysts quietly lose their integrity, almost always without meaning to.

A test finishes. The variant is down. Someone notices a single £4,000 trade order in the control. Removing it flips the result. Now what?

Whatever you do next, the damage is already done, because the decision to consider removing it was made after seeing which arm it landed in. That is p-hacking, regardless of intent — see pitfalls and validity threats.

The fix is procedural, not statistical. Before the test starts, write down:

  1. The rule. Winsorise at the 99th percentile, or trim above a fixed value, or cap quantity per order.
  2. That it applies identically to both arms.
  3. That the analysis will be reported both ways — capped and uncapped — so a result that depends entirely on the rule is visible as such.

Capping at p99 is a reasonable default for revenue metrics. It keeps the shape of ordinary behaviour and stops one whale from deciding a business question. If capped and uncapped disagree, you have not got a result; you have got one customer.

The distributions worth recognising

You do not need to derive these. You need to recognise which one you are looking at, because it tells you which tools apply.

  • Binomial — successes in n independent trials. Every conversion-rate test. Variance is np(1−p).
  • Poisson — counts in a fixed interval, where mean equals variance. Pageviews per session, tickets per day. When variance clearly exceeds the mean — which is common — you have over-dispersion, and negative binomial fits better.
  • Lognormal — right-skewed, strictly positive. Order values, durations. Log-transform makes it tractable, but be clear that you are then making claims about geometric means, which is not what the finance team means by “average order value”.
  • Heavy-tailed generally — where a meaningful share of total variance sits in a handful of observations. This is the regime where means are unstable and medians or capped means are more honest.

Percentiles beat averages for experience metrics

For anything users experience directly — page load, API latency, time to first result — the average is close to useless. Users do not experience your mean; they experience their own session.

Report p75 and p95. A page with a 1.2s average and a 6s p95 is failing one user in twenty badly enough for them to leave, and the average will never show it. This is also why Core Web Vitals thresholds are defined at the 75th percentile rather than the mean.

A pre-analysis checklist

Before running any test on a metric, know:

  • Its distribution shape — plot it, do not assume
  • Its standard deviation, so you can compute sample size
  • Its CV, so you know whether it is affordable to test on
  • Your outlier rule, written down and agreed
  • Whether the mean or a percentile is the right summary
  • Whether it is a ratio of two random quantities, which needs the delta method rather than a naïve proportion test

Where this connects

Descriptive statistics is the step everyone skips and then pays for later. Every sample-size calculation depends on a variance estimate; every choice of test depends on a distribution shape.