The Product Analytics & Data Science Cheat Sheet

DAU / WAU / MAU

Unique active users in a rolling 1, 7 or 28-day window. "Active" is a definition you choose — logging in is not the same as doing the thing the product is for.

Watch Windowed uniques don't add up: MAU ≠ sum of DAU. Never divide one by a differently-deduplicated denominator.

Stickiness DAU/MAU

Share of your monthly base that shows up on an average day. A proxy for habit: 0.2 is a monthly tool, 0.6 is a daily one.

Retention curve

Share of a cohort still active n periods after signup. Three flavours: N-day (active exactly on day n), unbounded (active on day n or later), rolling/bracket (active anywhere in a window around n).

Watch Unbounded retention is always the flattering one. State the flavour or the number is meaningless.

Cohort

A group defined by a shared start event — usually acquisition month. Cohorting separates genuine product improvement from a change in the mix of who arrived.

Churn rate

Share of active users or subscribers lost in a period. Logo churn counts accounts; revenue churn counts money; net revenue churn nets off expansion and can go negative.

Activation

The first moment a user gets real value — the event that best predicts week-4 retention. Found empirically, not decided in a workshop.

Time to value TTV

Elapsed time from first touch to the activation event. Report the median, not the mean — the distribution has a long tail of people who came back weeks later.

North star metric

The single number that best represents delivered customer value and leads revenue. One per product, movable by teams, hard to game.

OEC

Overall Evaluation Criterion — the metric an experiment is actually judged on, agreed before it starts. Often a weighted combination that trades short-term conversion against long-term retention.

Pirate metrics AARRR

Acquisition → Activation → Retention → Referral → Revenue. A coverage checklist, not a funnel: use it to notice which stage has no owner.

Session

A bounded run of activity by one visitor, cut after ~30 minutes idle, at midnight, or on a new campaign source depending on the tool.

Watch Session rules differ per platform, so session-denominated rates are not comparable across GA4, a warehouse model, and your test tool.

Bounce rate

Share of sessions with no qualifying engagement. In GA4 it is simply the inverse of engaged sessions (10s, a conversion, or 2+ views) — a different definition from Universal Analytics.

Exit rate

Of all views of a page, the share that were the last in the session. Unlike bounce, it counts pages deep in a journey — useful for finding the leak.

Conversion rate CR

Converting units ÷ exposed units. The denominator is the decision: sessions, users, or triggered-eligible users all give different — and differently sensitive — answers.

Funnel & step drop-off

Ordered stages with a pass rate at each. Choose closed (must enter at step 1, strict order) or open (enter anywhere) funnels deliberately.

Micro vs macro conversion

Macro is the money event; micro is a predictive precursor (size selected, delivery tab opened). Micro metrics move faster and are useful diagnostics — they are not a substitute for the macro result.

Average order value AOV

Revenue ÷ orders. Heavy-tailed and noisy — a single trade order can swing it, so it is a poor primary metric on low-volume tests.

Revenue per visitor RPV

Revenue ÷ sessions. The honest ecommerce OEC because it captures conversion and basket size together, and can't be won by pushing one at the other's expense.

ARPU / ARPPU

Revenue per user vs revenue per paying user. Moving ARPU can mean better monetisation or simply fewer free users — always read them as a pair.

Lifetime value LTV

Discounted gross profit expected from a customer. Use margin, not revenue: an LTV built on top-line makes low-margin categories look like winners.

CAC & payback

Fully-loaded acquisition spend ÷ new customers, and the months of contribution margin needed to repay it. Blended CAC hides that organic is subsidising paid.

Feature adoption

Breadth (share who ever used it), depth (uses per adopter), and duration (still using at day 30). Reporting breadth alone hides a feature everybody tried once.

Event vs pageview

Modern schemas are event-only: a pageview is just an event with a page name. Design an event taxonomy (name, properties, when it fires) before instrumenting, or you'll be renaming things in the warehouse forever.

Identity stitching

Joining anonymous device IDs to a known user ID after login, and backfilling earlier activity. Determines whether a cross-device journey looks like one person or four.

Attribution model

The rule that assigns credit for a conversion across touchpoints: last-click, first-click, linear, time-decay, position-based, or data-driven.

Watch Every model is a heuristic for correlation. None of them measure incrementality.

Last non-direct click

The industry default: credit the last channel that wasn't direct, within a lookback window. Structurally over-credits bottom-funnel channels — brand search and retargeting.

Data-driven attribution DDA

Credit distributed by a model (often Shapley-style) comparing converting and non-converting paths. Better than last-click; still observational, still not causal.

Marketing mix modelling MMM

Top-down regression of aggregate outcomes on spend, seasonality and price, with adstock and saturation terms. Privacy-proof and channel-complete, but low-resolution and prone to collinearity.

Incrementality

The conversions that would not have happened without the treatment. The only number worth spending against, and it is always lower than attributed conversions.

Holdout / ghost ads

A randomly withheld slice that never sees the campaign or feature, measured against the treated group. The clean way to get incrementality; costs you the revenue from the holdout.

View-through conversion

Conversion credited after an ad impression with no click. Almost entirely a measure of who the platform chose to show the ad to.

Watch Treat view-through as an audience-quality signal, never as evidence of effect.

Assisted conversion

Touchpoints on a converting path that weren't the final click. Useful for spotting starved upper-funnel channels; not additive with last-click totals.

Cart / checkout abandonment

Share of carts or initiated checkouts not completed. Separate the two: cart abandonment is largely browsing behaviour, checkout abandonment is a usability and trust problem you can fix.

Sampling

Reporting on a subset and scaling up. Introduces variance exactly where you're slicing thin, so segment-level readings from sampled reports are unreliable.

Bot & internal filtering

Excluding crawlers, uptime monitors, scrapers and office IPs. Unfiltered bot traffic inflates sessions, deflates conversion rate, and adds pure noise to every test.

Consent & modelled conversions

With consent mode, denied users send cookieless pings and the platform models the gap. Modelled numbers are estimates with their own error — don't reconcile them to the penny against your backend.

Server-side vs client-side

Client tagging is blocked by ad blockers, ITP and network loss (typically 10–30%); server-side is complete but blind to on-page behaviour. Most stacks need both, joined on a shared ID.

Data layer

A structured page-level object the tag manager reads from, decoupling analytics from DOM scraping. Scraped selectors break silently on every redesign.

Mean / median / mode

Centre of a distribution. On revenue and time-on-page — both right-skewed — the mean sits well above the median and moves with outliers.

Variance & standard deviation

Average squared deviation from the mean, and its square root in the original units. Variance is what determines how much traffic you need.

Standard error SE

Standard deviation of the sampling distribution of a statistic — how much your estimate would wobble across repeat samples. Shrinks with √n, which is why halving the interval costs 4× the traffic.

Coefficient of variation CV

σ/μ — relative noisiness. Compare metrics: revenue per session often has CV > 4, conversion rate under 1, which is why one needs far more traffic than the other.

Percentile / quantile

Value below which a given share of observations fall. Latency and page-speed work lives at p75 and p95; the mean hides the tail that users actually complain about.

Interquartile range IQR

p75 − p25. A spread measure that ignores tails, and the basis of the classic outlier fence at 1.5 × IQR beyond the quartiles.

Skew

Asymmetry. Positive (right) skew — a few very large values — is the default for revenue, order value and session duration.

Kurtosis & heavy tails

How much of the variance lives in extreme values. Heavy tails break the normal approximation at small n and make means unstable between reads.

Outlier handling

Cap (winsorise) at a fixed percentile such as p99, or trim entirely. Decide the rule before you see the data, apply it identically to both arms, and record it.

Watch Capping after peeking at which arm the whale landed in is p-hacking with extra steps.

Normal distribution

Symmetric bell, defined by μ and σ. Rarely the shape of your raw metric, but usually the shape of your mean — which is what the test needs.

Binomial distribution

Count of successes in n independent trials at probability p. The model behind every conversion-rate test.

Poisson distribution

Counts of events in a fixed interval where mean = variance. Fits pageviews per session or support tickets per day; over-dispersed data needs negative binomial instead.

Lognormal distribution

Right-skewed, non-negative — the natural shape for order values and durations. Log-transforming makes it tractable, but the result is then about geometric means.

Central limit theorem CLT

Sample means tend to normal as n grows, whatever the underlying shape. The licence for using z and t tests on ugly metrics — provided n is large and the tail isn't extreme.

Law of large numbers

Sample means converge to the true mean as n grows. Explains why day-one results swing wildly and then settle — and why acting on day one is gambling.

Correlation

Pearson's r captures linear association; Spearman's ρ captures monotonic association on ranks and survives skew and outliers better.

Watch r near zero rules out a straight line, not a relationship.

Null & alternative hypothesis H₀ / H₁

H₀: no difference between arms. H₁: there is one. A test only ever gives you grounds to reject H₀ — never to prove it.

p-value

Probability of seeing a difference at least this extreme if the null were true. Not the probability the variant is better, and not the probability you're wrong.

Significance level α

Your pre-committed false-positive rate, conventionally 0.05. Choose it for the cost of the decision: a risky platform change deserves 0.01, a copy tweak can live at 0.10.

Type I error

Rejecting a true null — shipping a change that does nothing. Its long-run rate is α, but only if you tested once, on one metric, at a fixed sample size.

Type II error β

Failing to detect a real effect. The invisible error: nobody writes a post-mortem about the winner they binned as "flat".

Statistical power 1 − β

Probability of detecting an effect of a given size if it's real. 80% is the convention — meaning one in five true winners is missed.

Minimum detectable effect MDE

The smallest true lift the test can reliably detect at your α, power and sample size. Set it from what would be worth shipping, then check you can afford the traffic.

Confidence interval CI

A range built so that 95% of such intervals, across repeated experiments, would contain the true effect. Report the interval, not the point estimate — its width is your real answer.

Credible interval

The Bayesian counterpart: given the model and prior, a 95% probability the true effect lies inside. This is the interpretation people wrongly assume a CI has.

One- vs two-tailed

One-tailed tests only for improvement and is cheaper; two-tailed can also detect harm. Web experiments break things often enough that two-tailed is the honest default.

z-test of proportions

The standard conversion-rate test: normal approximation to the difference of two proportions. Reliable once each arm has roughly 30+ conversions.

t-test / Welch's t-test

Compares two means. Always prefer Welch's — it doesn't assume equal variances, and treatment usually changes spread as well as centre.

Chi-square test

Compares observed against expected counts across categories. Used for k-arm conversion tables and, critically, for sample ratio mismatch checks.

Fisher's exact test

Exact p-value for a 2×2 table. Reach for it with very low counts, where the normal approximation misbehaves.

Mann–Whitney U

Rank-based comparison of two distributions, robust to skew and outliers. It tests stochastic dominance, not means — so it can disagree with the revenue figure the business cares about.

Bootstrap

Resample the data with replacement thousands of times to build an empirical sampling distribution. The general-purpose escape hatch for metrics with no clean closed form.

Permutation test

Shuffle the arm labels repeatedly to construct the null distribution directly. Assumption-light and intuitive; costs compute, not statistical knowledge.

Delta method

Variance approximation for ratio metrics whose numerator and denominator are both random — revenue per user, items per order.

Watch Naïvely applying a proportion test to a ratio metric understates variance and manufactures significance.

Effect size / Cohen's d

Magnitude of a difference in standard-deviation units, independent of sample size. Guards against the large-sample trap where everything is significant and nothing matters.

Absolute vs relative lift

2.0% → 2.4% is +0.4pp absolute and +20% relative. Power calculations take absolute; stakeholders hear relative. Label which one you mean, every time.

Practical significance

Whether the effect is large enough to be worth the cost of shipping and maintaining. A significant +0.2% on a low-traffic page may never repay the code.

Rule Decide the practical threshold before the test, alongside the MDE.

Wilson score interval

A better interval for proportions than the textbook normal one, especially at small n or rates near 0 or 1, where the naïve interval can extend past the boundary.

A/B test

A randomised controlled trial on live traffic: units are assigned at random to control or variant, so the only systematic difference between arms is the change itself.

A/A test

Both arms identical. Validates the platform end to end — randomisation, tracking, variance estimates. Roughly 5% of A/A tests should still show "significance" at α = 0.05; if far more do, something is broken.

A/B/n

One control against several variants. Each extra comparison inflates the family-wise false-positive rate and splits your traffic, so power falls twice over.

Multivariate test MVT

Multiple factors tested simultaneously to estimate main effects and interactions. Traffic-hungry — usually only worth it when you genuinely suspect an interaction.

Split URL test

Traffic redirected to a separate page or template. Fine for wholesale redesigns; the redirect adds latency and can leak the test through the address bar.

Randomisation unit

The thing assigned to an arm — visitor, user, session, or account. It must match the analysis unit, and users must stay in the same arm for the whole test.

Watch Session-level randomisation on a multi-visit journey reshuffles people mid-funnel and destroys the comparison.

Bucketing / hashing

Deterministic assignment by hashing the unit ID with an experiment salt. Gives stable, reproducible, independent allocation across concurrent tests.

Sample size calculation

Traffic needed per arm for a given α, power and MDE — computed before launch, never after. The quick approximations:

Note The 16 encodes α = 0.05 two-sided and 80% power. Δ is the absolute difference.

Runtime & business cycle

Run at least one full weekly cycle, usually two, even if the sample target lands sooner. Weekday and weekend visitors are different populations.

Trigger / exposure analysis

Count only users who actually reached the changed element. Including everyone dilutes a real effect with people who could never have been affected.

Intention to treat ITT

Analyse users by assigned arm regardless of whether treatment rendered. Conservative but unbiased; analysing only the compliant reintroduces selection bias.

Guardrail metric

A metric that must not degrade even if the primary wins — page latency, error rate, refunds, unsubscribes, margin. Defined and monitored from day one.

Primary vs secondary metrics

One primary metric decides the test. Secondaries explain the mechanism and are read descriptively, without significance claims, unless you correct for them.

Metric sensitivity

How readily a metric moves when the product genuinely improves. Revenue is the metric you care about; a sensitive upstream proxy is often the one you can actually power.

CUPED

Controlled-experiment Using Pre-Experiment Data: subtract a pre-period covariate to strip out variance that has nothing to do with treatment. Frequently cuts required runtime by 30–50%.

Stratification & blocking

Randomise within strata (device, country, new vs returning) so arms are balanced on known drivers. Cheap variance reduction, and it makes segment reads more trustworthy.

Cluster randomisation

Assign whole groups — accounts, stores, geos — when individuals inside a group interact. Analysis must use the cluster as the unit, or intervals come out far too narrow.

Switchback test

Alternate the whole system between control and treatment in time slices. The standard tool for marketplaces and logistics where user-level splits contaminate each other.

Holdback group

A small slice kept on control for weeks after rollout, to measure whether the day-14 lift survives to day 90.

Feature flag

Server-side toggle controlling exposure and allocation. Avoids client-side flicker, supports staged rollout, and gives an instant kill switch.

Multi-armed bandit MAB

Adaptively shifts traffic toward the arm performing best, trading measurement precision for less regret. Right for short-lived allocation decisions; wrong when you need a clean effect estimate.

Thompson sampling

Bandit policy that samples from each arm's posterior and plays the winner. Balances exploration and exploitation without a hand-tuned ε.

Sequential testing

Designs — mSPRT, group-sequential, always-valid confidence sequences — that permit continuous monitoring while keeping the false-positive rate controlled. The legitimate answer to "can we stop early?".

Alpha spending

Budgeting α across planned interim looks (O'Brien–Fleming, Pocock) so the total stays at 0.05. Early looks get a very strict threshold.

Bayesian A/B testing

Combine prior with data to get a posterior over the effect, and report probability-to-beat-control and expected loss. Directly answers the business question; the prior is a real assumption, not a formality.

Expected loss

Average cost of shipping the wrong arm, under the posterior. Ship when expected loss falls below a threshold you set — a cleaner stopping rule than chasing 95%.

Interaction between tests

Concurrent experiments on the same surface can interfere. Independent hashing keeps allocation orthogonal; genuinely conflicting changes need mutual exclusion.

Peeking

Repeatedly checking and stopping the moment p < 0.05. With enough looks the false-positive rate climbs toward 30%+ under a fixed-horizon test.

Fix Fix the sample size in advance, or adopt a sequential method built for monitoring.

Multiple comparisons

Testing many metrics, arms or segments. At α = 0.05, 20 independent comparisons yield roughly one false winner by chance alone.

Bonferroni correction

Test each of m comparisons at α/m. Simple and strict — it protects the family-wise error rate at a real cost in power.

Benjamini–Hochberg FDR

Rank p-values and compare each to (i/m)·q, controlling the expected share of false discoveries. The practical choice when scanning many metrics.

Sample ratio mismatch SRM

Observed split departs from the intended one (say 52/48 on a planned 50/50). A chi-square p below 0.001 means assignment or logging is broken.

Fix Stop and diagnose. An SRM test result is not salvageable by analysis.

Simpson's paradox

A trend present in every segment reverses when segments are pooled, because the arms have different segment mixes. Classic on device or geo splits.

Survivorship bias

Analysing only the units that made it — completed sessions, retained users — and inferring something about everyone. Systematically flatters the treatment.

Selection bias

Any process that makes exposure non-random: opt-in betas, one arm loading slower, a redirect that mobile Safari drops. Randomisation is the only defence, and it must actually hold.

Novelty effect

Returning users engage with a change because it's new. Shows as a strong early lift that decays. Read the effect by cohort week, and check new users separately.

Primacy effect

The mirror image: regular users are disrupted by a change and perform worse until they relearn. Kill a UI change on day-three data and you may be killing a winner.

Regression to the mean

Extreme observations drift back toward average on remeasurement. Optimising the pages that looked worst last month will "work" even if you change nothing.

Twyman's law

Any figure that looks interesting or unusual is usually wrong. A +40% lift on a checkout tweak is a tracking bug until proven otherwise.

p-hacking

Trying variations of the analysis — filters, outlier rules, windows, segments — until something clears 0.05. Rarely deliberate; almost always what happened when a flat test becomes a win.

HARKing

Hypothesising After the Results are Known, then presenting it as the original prediction. Turns exploratory noise into a confirmed finding.

Segment mining

A flat test "wins on mobile in Germany for returning users". Post-hoc segments are hypotheses for the next test, never conclusions from this one.

Flicker FOUC

Control renders before the client-side variant swaps in. Visible to users, biases the variant downward, and is worst on slow connections. Server-side rendering removes it.

Instrumentation bias

The variant's tracking fires differently — an extra script, a changed selector, a slower load. The measured difference is then partly measurement, not behaviour.

Cross-contamination

Users seeing both arms via shared devices, cleared cookies, cached pages or CDN mixing. Dilutes true effects toward zero.

Network interference SUTVA

One unit's treatment affects another's outcome — shared inventory, social feeds, marketplace supply. Breaks the core assumption of independence; needs cluster or switchback designs.

Seasonality

Payday, term dates, weather, promotions. Randomisation handles it within a test window, but it wrecks any before-and-after comparison.

Internal vs external validity

Internal: the effect is real for the tested population. External: it generalises to other traffic, seasons and markets. A clean test on Black Friday traffic has both properties in different amounts.

Heterogeneous treatment effects HTE

The average effect masks winners and losers inside it. Pre-registered segments and uplift models surface this honestly; fishing does not.

Winner's curse

Effects that just cleared the significance bar are biased upward, because only large sampled effects get through. Expect measured lifts to shrink on rollout, especially from underpowered tests.

Potential outcomes

Each unit has an outcome under treatment and under control; only one is ever observed. Causal inference is the business of estimating the missing half.

Counterfactual

What would have happened without the change. Every claim of impact is a claim about a counterfactual, whether or not it's stated.

Confounder

A variable driving both treatment and outcome — a promo running during a release, seasonality during a redesign. Randomisation balances confounders you never thought of; adjustment only handles the ones you measured.

Quasi-experiment

Exploiting a naturally occurring split — a phased rollout, a regional launch — as if it were assigned. Credible only if you can argue exposure was as-good-as-random.

Difference-in-differences DiD

Compare the before→after change in a treated group to the same change in an untreated one, cancelling out common time trends.

Watch Requires parallel pre-trends. Always plot them.

Synthetic control

Build a weighted composite of untreated units that reproduces the treated unit's pre-period, then read the post-period gap. The workhorse for single-market or single-site interventions.

Regression discontinuity RDD

Compare units just either side of a threshold — free-shipping minimums, loyalty tiers, ranking cut-offs. Local to the boundary, but very credible there.

Instrumental variable IV

A variable affecting treatment but not the outcome except through treatment. Recovers effects under non-compliance; valid instruments are rare and the assumption is untestable.

Propensity score matching

Model probability of treatment from covariates, then compare like with like. Balances only what you observed — it cannot fix unobserved confounding.

Geo-lift test

Randomise or match at region level and compare treated to untreated geos. Standard for measuring media incrementality where user-level holdouts aren't possible.

Linear regression

Fits a continuous outcome as a weighted sum of inputs. In experimentation it doubles as a variance-reduction tool: regress the outcome on pre-period covariates.

Logistic regression

Models a binary outcome; coefficients are log-odds, exponentiated into odds ratios. Interpretable, well-calibrated, and still the sensible baseline for conversion or churn.

Precision & recall

Precision: of those flagged, how many were right. Recall: of the real cases, how many you caught. The trade-off is set by your threshold, not by the model.

F1 score

Harmonic mean of precision and recall. A single number for balanced comparison — but it hides which of the two errors you're actually making.

ROC-AUC

Probability the model ranks a random positive above a random negative. Threshold-free and readable, but flattering on heavily imbalanced data.

PR-AUC

Area under the precision–recall curve. The right headline metric when positives are rare — churn, fraud, high-intent visitors.

Calibration

Whether a predicted 0.3 actually converts 30% of the time. Essential whenever the score feeds pricing, bidding or expected-value logic. Check with a reliability plot.

Overfitting

Learning noise specific to the training set. Detected by a train/validation gap; controlled with regularisation, simpler models, or more data.

Cross-validation

Rotate held-out folds to estimate out-of-sample performance. For anything time-dependent use forward-chaining splits — random folds leak the future into the past.

Uplift modelling CATE

Predicts the incremental effect of treating a given user, not their outcome. Separates the persuadables from the sure things — the difference between a discount that works and one you gave away.

Propensity model

Scores likelihood of an action (purchase, churn, upgrade). Powers targeting and triggers; recalibrate on a schedule as behaviour drifts.

RFM segmentation

Recency, Frequency, Monetary scoring. Crude, ancient, and still one of the strongest retail segmentations per hour of effort.

Clustering

Unsupervised grouping — k-means on scaled features, or hierarchical for structure. Segments are hypotheses; validate that they differ on outcomes you care about.

Survival analysis

Time-to-event modelling (Kaplan–Meier, Cox) that handles censoring — users who haven't churned yet. Correct where a simple churn rate throws away information.

Time-series decomposition

Split a series into trend, seasonality and residual. The residual is where anomalies and genuine interventions live.

Anomaly detection

Flagging departures from expected behaviour, seasonally adjusted. Most production value is in catching broken tracking within hours instead of at month end.

SHAP values

Per-prediction feature attribution from cooperative game theory. Explains what drove a score — an association, not a lever you can pull.