Our Thoughts

When A/B Tests Lead You Astray

17th February 2020

I'm a big believer in experiments, or A/B tests as they're known in marketing and UX.

Although I make my living conducting research with customers, like any honest researcher I'd be happy to admit that talking to customers is a very bad way of predicting their behaviour.

Experiments are useful because, if we take a bit of care with the design, they reliably reveal cause and effect relationships. That means we can use them to draw cast-iron conclusions about what would happen if we roll out (for example) a new process, or a redesigned website, to the whole customer base.

Experiments reveal the effect of one variable on another, for example if we A/B test two email subject lines we can measure the impact of the subject line on response rates. But there's an important catch: experiments only tell us about outcomes that are included in the data.

Have you measured what really matters?
It's a surprisingly difficult question: are you measuring what really matters? A common criticism of drug trials is that the outcome variable may be a "surrogate endpoint", rather than the thing we're really interested in. We might show, for instance, that a new drug helps to reduce cholesterol, but interpreting that as reducing the risk of death from heart disease may be problematic. Exactly the same is true with customer data – if you don't include the data that really counts in the experiment, then you can reach dangerously flawed conclusions.

An example may help to make it clear. Let's say you run a web shop and you trial offering free shipping for orders over £20. As you hoped, customers with baskets just under £20 find add-ons to take them over the threshold, and your average basket spend increases. Great, you're making more money. Or are you? Average basket spend is only one potential outcome. Maybe if you looked at abandonment rates, you'd find that the new rule was actually annoying some customers and driving them away. Or maybe it has a positive impact on short term spend, but makes customers less likely to return to your store in the future?

If you only look at data that has a short term view, you risk making decisions which prioritise the short term over the long term. It'll lead you, for instance, to use pop-ups inviting customers to sign up to your newsletter, even though everyone hates them.

As we've seen, customers are not very good at predicting, or even remembering, their own behaviour, but what they are good at is telling you how they think and feel. With intelligent interpretation that means that you can uncover their needs (both functional and emotional).

Getting the balance right

By using the right combination of experiments and research, combining "big data" with "small data", you can develop a rounded picture of customers that will enable you to create customer experiences which are more effective and more in tune with customers. So what's the takeaway?

  • Use experiments to understand the drivers of behaviour, but try to make sure you have the data that really matters (which is normally customers' long term behaviour)
  • Turn back to research to undertand customers' needs and attitudes
01484 517575
Taylor Hill Mill, Huddersfield HD4 6JA
Twitter LinkedIn