I first remember personalisation being the main conference topic in about 2006. It has been the main conference topic most years since. Twenty years is a long time to be on the verge of something.
In that time a fair number of companies have reached a basic level of it. A different homepage for a logged-in customer. A recommendation carousel. An email that knows what you bought. Very few have got anywhere near what the keynotes described, where every visitor gets an experience assembled for them alone. “Hyper-personalised” still mostly belongs in inverted commas.
Now the keynotes have a new argument: AI changes everything. For once I think the claim deserves a proper look, because AI does remove one of the two things that held personalisation back. It leaves the other one exactly where it was.
Here is the picture I use to explain both.
Schematic, not data. Each step to the right cuts the audience into smaller groups, so each new experience reaches fewer people while the number of experiences to build, check and measure multiplies. The dashed line is the question this article asks.
Moving left to right, the audience is cut into smaller and smaller groups. Two things follow. The work multiplies, because every group needs its own experience, and somebody has to build, check and measure each one. And the return from each new experience shrinks, because it reaches fewer people. For twenty years those two lines have crossed somewhere around the second stage, and that is where most programmes stopped.
The question is whether AI moves the red line. I think it moves half of it.
Four stages, left to right
The diagram above has four columns. They are not a maturity model in the sense that everyone should be marching to the right. Each is a legitimate place to stop, and for plenty of businesses the first or second is the correct one.
Stage 1: one size fits all
Everyone gets the same website or app. It sounds primitive and it has two large advantages. Every improvement reaches the whole audience, and every measurement is a clean comparison between two groups drawn from all of your traffic.
It is also where the largest, cheapest wins are, because a broken checkout is broken for everybody.
Stage 2: persona-based
The audience is split into a handful of groups that are decided in advance: new and returning, business and leisure, trade and retail, app and web. Each group gets its own version of the important pages.
This is where many companies have got to, and it works for an unglamorous reason. The groups are big. A persona that makes up a fifth of your traffic is large enough to build for, large enough to test on and large enough to show up in the revenue line when you get it right.
Stage 3: behavioural
Segments are defined by what people do, not who they are assumed to be. Viewed three products in a category without adding one. Searched the same route twice this week. Abandoned at delivery options. There are dozens of these, they overlap, and people move between them from one session to the next.
A few organisations run this well. They tend to be the ones with a large engineering team, a mature data platform and enough traffic that even a narrow segment is a big number.
Stage 4: hyper-personalised
A segment of one. The experience is assembled for the individual, at the moment they arrive, from everything known about them. No two people need see the same thing.
This is the stage that has been promised since 2006, and in my opinion it has mostly remained a slide. There are products that feel like it: a video or music feed, a recommendations row. Look closely and those are one fixed layout with a ranked list inside it. The list is personal. The experience around it is stage one.
What happens as you move right
The same two things happen at every step.
Each experience reaches fewer people. A 10% lift for a segment that is 2% of your traffic is a 0.2% lift for the business. The segment has to be valuable out of all proportion to its size before that pays for the work.
The number of things to build and measure multiplies. One experience becomes five, then fifty. Then personalise four parts of a page with five options each and there are 625 possible pages, which nobody has reviewed and nobody can test one at a time.
Those are the blue and red lines on the diagram. Everything else in this article is about what does and does not move them.
Why twenty years of trying got most companies to stage two
The usual explanation is that the technology was not ready. I do not think that is right. The decisioning engines existed, the customer data platforms existed, and the vendors were very willing to sell both. Personalisation stalled on four ordinary problems, and it is worth being exact about them, because AI only helps with some.
Somebody has to make all the versions
Every segment needs something different to look at. That means copy, imagery, layout, offers and a sign-off for each one. A team that struggles to keep one homepage current does not have the capacity to keep fifty current. The supply of content tends to run out long before the supply of segments does.
The return gets diluted
This is the blue line in the four-stage diagram. A new experience for a narrow segment can work very well for the people in it and still be invisible in the business’s numbers, because there are so few of them. The work to build it is roughly the same as the work to build something for everyone.
The clearest example is from the year it all started
In October 2006 Netflix offered a million dollars to anyone who could improve the accuracy of its recommendations by 10%. It took three years, and a team called BellKor’s Pragmatic Chaos won in 2009.
Netflix never put the winning model into production. Its engineers wrote in 2012 that the additional accuracy “did not seem to justify the engineering effort needed to bring them into a production environment”. The most famous personalisation project in history was won, paid for and shelved, by the company that commissioned it, on a complexity-against-return calculation.
If Netflix could not make the last step pay with that much traffic and that much talent, it is not surprising that a mid-sized retailer could not either.
Nobody could tell whether it was working
This is the problem that gets the least attention, and it is the one I care most about.
Take a site with a million visitors a month and a 2% conversion rate. To detect a 10% relative lift at the usual settings you need roughly 80,000 visitors in each arm of a test, so 160,000 for a simple A/B.
- At stage one, that test finishes in about five days.
- At stage two, with five personas of 200,000 visitors each, each persona can conclude a test in under a month.
- At stage three, with fifty behavioural segments of 20,000 each, a single test in a single segment takes eight months.
- At stage four there is no segment to test in at all.
Then there is the reading of the results. Fifty segments, each checked at the usual 5% threshold, gives a 92% chance that at least one of them shows a “significant” effect when nothing has changed. That is the multiple comparisons problem, and a personalisation programme produces it by design.
So the typical programme ended up in an uncomfortable place. It could not show that the behavioural segments were paying, it could show what they cost, and at the next budget round it quietly went back to personas.
What AI moves, and what it leaves alone
The four-stage diagram at the top of this article has a red line for complexity that climbs steeply to the right, and a dashed green line asking whether AI flattens it. My answer is that the red line is really two costs added together: the cost of building the experiences and the cost of knowing whether they worked. AI flattens the first. It does very little for the second, and in one respect makes it worse.
What it changes
The cost of a variant. This was the first thing to run out in every personalisation programme: nobody could produce fifty versions of anything. A language model can produce fifty versions of a headline, a product description or an email before the kettle has boiled, and adapt each to a segment. The content supply problem, which held things at stage two for many teams, has largely gone.
The segments no longer have to be defined in advance. Behavioural targeting used to mean a rules engine: if they did this and this, show that. Someone had to think of every rule. A model can take signals that never fitted in a rule, such as a search phrase, the order someone looked at things in, or what they typed into a chat box, and respond to them directly.
The interface itself. A conversational or agent-driven interface is personalised by construction. The answer is composed for the question that was asked. Nobody had to design a segment for “family of four, flexible on dates, one of them hates connections”. That is new in kind, and it is the strongest reason to think stage four is finally reachable.
The conversation has shifted where I work as well. In a large product organisation, hyper-personalisation is being talked about a great deal more than it was, and as something to scope, not something to put on a slide.
What it does not change
The arithmetic of small groups. A segment of 20,000 visitors is still 20,000 visitors. The blue line on the diagram is not a technology problem and no model moves it. A wonderful experience for very few people is still a small number.
The need to know whether it worked. A generated experience that has not been measured is an opinion delivered at scale. The eight months it takes to conclude a test in a behavioural segment, described in the “Why It Stalled” tab, is the same eight months whoever wrote the variant.
The quality of the data underneath. A model personalising on behavioural signals believes those signals. If “add to basket” fires twice on one template and never on another, the personalisation will be confidently wrong for the affected customers, and tidy-looking while it does it. I have written about what a bad event taxonomy costs before. Personalisation raises the price.
What it makes harder
Nobody has seen what the customer saw. At stage two a person signed off five versions of the page. At stage four there may be a different page for every visitor, and none of them were reviewed. When one of them is wrong, misleading or just odd, the only person who knows is the customer who got it.
The thing being tested stops being a page. With a fixed variant you can say what the treatment was. With a generated experience the treatment is the system that produces it: the model, the prompt, the data it was given. Change any of the three and it is a different treatment. The analysis has to compare policies, not pages, and most analytics set-ups were not built for that.
Automatic optimisation has its own traps. Many hyper-personalisation products pair a model with a bandit that shifts traffic towards whatever seems to be working. That goes wrong in a specific way when customers take days to convert, which I worked through in multi-armed bandits and return-user bias.
What I do not know
I have not seen a hyper-personalised experience measured against a proper holdout, with a result I would defend in front of a finance director. There are plenty of case studies. Almost all of them compare the personalised experience with what the same customers did before, or add up lifts from individual segments, and neither is a measurement.
That may simply mean the good results have not been published. If you have one, I would like to see it, and I mean that as an invitation, not a challenge.
The constraint has moved to analytics
If AI takes away the cost of building, the organisations that get to stages three and four will be the ones that can tell whether it is paying. That is a measurement problem, and it can be solved. It needs four things to be in place before the personalisation is switched on, not after.
1. A holdout that never gets personalised
A slice of the audience, chosen at random and kept for months, that gets the stage-one experience. Everyone else gets whatever the personalisation system decides.
This turns an impossible question into an easy one. You cannot test fifty segments individually. You can compare everyone who was personalised with everyone who was not, using all of your traffic, and that comparison concludes in days. It tells you whether the system as a whole is earning its keep, which is the question the budget holder is asking.
2. A record of what each person was shown
Not which variant they were in, because there may not be one. What was generated or selected for them, from which inputs, by which version of the model and the prompt. Without that log, a bad outcome cannot be investigated and a good one cannot be explained or repeated.
It is also the record you will want on the day a customer complains about something only they saw.
3. Segment definitions written down before the results arrive
Behavioural segments are easy to define after the fact in whatever way makes the numbers look best. Decide in advance which segments count, what the success metric is and which guardrails block a win. Anything discovered afterwards is a hypothesis for the next test.
4. Checks on the output, run by a machine
If nobody reviews the individual experiences, something else has to. That means deciding what an acceptable output is, such as correct prices, real stock, nothing the brand would not say and nothing the regulator would object to, and testing the system against it continuously. I have argued that evals are the bottleneck for AI in analytics. They are the bottleneck here for the same reason.
What you get for it
A business with those four things can answer three questions that most personalisation programmes of the last twenty years could not:
- Is it paying? From the holdout, in money, for the whole system.
- Where is it paying? From pre-declared segments, read honestly.
- What went wrong for that customer? From the exposure log.
It can also move right along the diagram one step at a time and stop when the return no longer covers the work. For most businesses that stopping point will still be short of stage four. The difference is that it will be a decision made on evidence. Until now it has usually been exhaustion.
Who this is not for
- Sites that have not finished with stage one. If the checkout has obvious problems and the product pages have never been tested, fix those for everyone first. It is cheaper and the return is larger.
- Sites without the traffic. If a persona cannot conclude a test in a quarter, behavioural segments certainly cannot. The low-traffic playbook is a better use of the time.
- Businesses whose event data is not trusted. Personalising on signals nobody believes produces experiences nobody can explain.
- Anyone who needs it live by Christmas. The holdout takes months to say anything useful, so it has to start first.
How Conversion Works helps
Everything in this tab can be built in-house. The hard parts are the order, and being honest about the answer.
We start by finding out whether you should be doing this at all. How big your candidate segments are, how long a test would take in each, what your event data can and cannot support, and where on the four-stage diagram the return stops covering the work for your business. Quite often the answer is “stage two, done properly”. That is worth knowing before a vendor contract is signed, and the piece of work is useful on its own.
Then we build the measurement before the personalisation. The holdout, the exposure log, the written segment definitions and the output checks. It is handed over as something your team runs and owns.
Then we measure whether it paid. Against the holdout, in margin, over long enough to matter. It is the step that was skipped for twenty years.
Behind that is twenty-plus years in experimentation and analytics, more than 6,000 experiments and two production products built for this work.
The point
Personalisation did not stall for twenty years because nobody could build a decisioning engine. It stalled because every step towards the individual made the work multiply and the audience shrink, and somewhere around personas the two lines crossed.
AI moves one of them. Producing a different experience for every segment, or every person, has gone from impossible to cheap, and conversational interfaces are personal without anyone designing a segment at all. I think that is enough to say the door to real personalisation is open in a way it has not been before.
It does not move the other. Small segments are still small, a test in one of them still takes months, and an experience nobody reviewed still has to be shown to be working. The complexity has not gone away. It has moved out of the content team and into analytics.
So my answer to the question in the title is a qualified yes. AI probably is the harbinger. Whether a given company gets the benefit depends on whether it can measure a system in which no two customers saw the same thing, and most cannot yet.
I would like to be shown I am being too cautious. If you have run hyper-personalisation against a real holdout, I would like to hear what happened, whichever way it went.
Two things have to be true before any of this is worth switching on. The behavioural signals have to mean what you think they mean, which is what Conform protects by enforcing event schemas at the edge. And the experience has to be decided before the page renders, with assignment that stays with the visitor so that a holdout stays held out, which is what Conversion Workers does. If you would rather start by finding out where on the diagram your return runs out, get in touch. That answer is worth having whatever you decide to build.