A churn model that is 94% accurate can be completely worthless. If 94% of your customers do not churn in a given month, a model that predicts “nobody churns” achieves exactly that accuracy and tells you nothing.

That is the whole problem with applying machine learning to product analytics: the modelling is the easy part, and the metrics everyone reaches for first are the ones least suited to the decisions being made.

This expands the modelling and prediction section of the cheat sheet.

Start with regression, and often stay there

Logistic regression remains the right first model for most product problems — conversion, churn, upgrade propensity. It is interpretable, it calibrates well, it trains in seconds, and it gives you coefficients you can argue about with a stakeholder.

Gradient boosting will usually beat it on raw discrimination. Whether that improvement is worth the interpretability and maintenance cost depends entirely on what the score is for. If it feeds an automated bidding system, take the accuracy. If it feeds a conversation with the marketing team about which customers to call, take the interpretability.

Linear regression has a second life in experimentation as a variance-reduction tool — regressing the outcome on pre-period covariates is the same idea as CUPED, covered in experiment design.

Classification metrics that survive imbalance

Product outcomes are usually rare. Churn in a month, fraud, high-intent visit — all small positive classes. That breaks the intuitive metrics.

Accuracy is useless under imbalance, for the reason in the opening paragraph.

Precision and recall are the working pair:

precision = TP / (TP + FP)      of those flagged, how many were right
recall    = TP / (TP + FN)      of the real cases, how many we caught

The trade-off between them is set by your threshold, not by the model. This is the most consequential thing to understand: one trained model gives you an entire family of precision/recall combinations, and choosing among them is a business decision about the relative cost of a false positive versus a false negative.

If a false positive means sending a £10 retention voucher to someone who was never going to leave, precision matters and the threshold should be high. If a false negative means losing a £2,000 annual account, recall matters and the threshold should be low.

F1 collapses both into one number, which is convenient for model comparison and hides which error you are making.

ROC-AUC — the probability the model ranks a random positive above a random negative — is threshold-free and easy to communicate. It is also flattering under heavy imbalance, because the huge negative class makes the false-positive rate look small.

PR-AUC is the honest headline when positives are rare. If your positive class is under about 10%, lead with PR-AUC and treat ROC-AUC as supporting.

Calibration is what most teams skip

A model is calibrated when a predicted probability of 0.3 corresponds to a 30% real-world rate.

Discrimination and calibration are different properties. A model can rank perfectly — every true positive above every true negative — while every predicted probability is wrong by a factor of three. If you are only picking the top N to target, ranking is enough. If the score feeds anything that treats it as a probability, calibration is essential:

  • Expected value calculations (P(convert) × margin)
  • Bid amounts
  • Discount sizing
  • Any threshold expressed as “customers over 40% risk”

Check it with a reliability plot: bucket predictions by score, plot mean predicted against actual rate, and look for the diagonal. Fix it with Platt scaling or isotonic regression on a held-out set. It is an afternoon’s work and it is routinely the difference between a model that makes money and one that quietly loses it.

Propensity is not the same as impact

Here is the mistake that costs the most money in retention and CRM.

You build a churn model. You take the top 10% highest-risk customers and send them a discount. Churn among the targeted group is high — the model was right about that. But some of those people were going to leave regardless of any voucher, and some of the ones who stayed would have stayed anyway. You spent budget on both.

Propensity models predict outcomes. Uplift models predict the effect of treating someone.

Uplift modelling — also called CATE estimation, for conditional average treatment effect — splits your audience into four groups:

  • Persuadables — buy only if treated. The only group worth spending on.
  • Sure things — buy regardless. Treating them is pure margin given away.
  • Lost causes — never buy. Treating them is wasted spend.
  • Sleeping dogs — treating them makes things worse. Real, and common in messaging: the reactivation email that reminds a dormant customer to cancel.

Building an uplift model requires experimental data — a randomised treated and untreated group — because you are modelling a treatment effect, and treatment effects only exist relative to a control. This is a genuinely important consequence: your experimentation programme is what makes your targeting models possible. Teams that never hold out a control group cannot build these at all.

The payoff is direct. Targeting persuadables rather than high-propensity customers typically improves campaign ROI substantially, largely by stopping spend on the sure things.

Survival analysis for retention

Standard churn modelling asks “will this customer churn in the next 30 days?” — a binary question over a fixed window. That throws away information, and it has a subtle bug: customers who have not churned yet are not the same as customers who did not churn.

Survival analysis handles this censoring properly. Kaplan–Meier curves give you retention over time without pretending recent signups have had a full chance to churn. Cox proportional hazards models tell you which covariates change the rate of churn rather than a binary probability.

Where this matters most is LTV. A retention curve modelled with survival methods gives a far more defensible lifetime value than dividing by a blended churn rate — particularly for products where the hazard is very high in the first period and flattens sharply afterwards, which is nearly all of them.

Segmentation that earns its keep

Clustering — k-means on scaled features, or hierarchical when structure matters — produces segments. Whether they are useful segments is a separate question that people forget to ask.

The test: do the clusters differ on outcomes you care about, and can you act differently toward each? A segmentation that splits users into groups with identical conversion rates and identical needs is a decorative exercise.

RFM — recency, frequency, monetary — is crude, decades old, and still one of the highest-return segmentations per hour of effort in retail. Start there before reaching for anything unsupervised.

Explanation, drift and honesty

SHAP values attribute a prediction to its features, which is genuinely useful for debugging a model and for explaining a score to a stakeholder. What they do not give you is causality. “Discount usage contributed +0.2 to this churn score” does not mean removing discounts reduces churn — that requires an experiment or a causal design.

Behaviour drifts. A propensity model trained on last year’s traffic degrades as your mix, product and market change. Monitor calibration on recent data, not just at training time, and set a retraining cadence before performance decay becomes visible in results.

Validate against time, not at random. For anything time-dependent — which is most product data — random cross-validation folds leak the future into the past and produce optimistic scores that never materialise. Use forward-chaining splits: train on earlier periods, test on later ones.

Where this connects