Learning from Bus Buddhists
In psychological terms, context is almost everything. Much as we like to think that we know how we will act and react in a given situation, without the richness of...
It’s always interesting to see what ideas a web giant like Amazon is evaluating, particularly when the change is more than just a subtle adjustment.
At present they’re trying out a very different home page.
One thing Amazon does that is super-smart is split test. They don’t rely on the vague irrelevancies of what customers tell them in market research to decide whether a change is worth making: they send customers to the new look at random and check to see what the impact is on behaviour (and particularly on conversion). This means the evaluation isn’t done as an artificial conscious exercise, people don’t know they’re taking part in research: as a result the unconscious drivers of consumer actions are still ‘in-play’ and the artificial influence that comes from asking questions isn’t an issue.
From this perspective it doesn’t really matter what I or anyone else thinks when they evaluate the new design. You may love its clean look, the absence of clutter, the way it works properly in a wide screen format, enjoy accessing menus when you choose (rather than having them forced on you)… but if you end up not searching so effectively or not clicking on the day’s promotions, Amazon will quietly revert to a more profitable previous look and go back to the drawing board.
Tiny changes can have a big impact on conversion. In Consumer.ology I recount a number of examples where changing a photograph, moving a logo or reducing the number of items returned from a search all increased conversion significantly. Whilst as a consumer psychologist I can help identify what might work, and can speculate on what has made a change effective by explaining how it has connected with unconscious drivers of behaviour, it’s only through carefully conducted experiments that we can know for sure what the outcome of a change will be.
Fortunately, split testing is relatively straight-forward on-line, it’s actually reasonably straight-forward in most consumer scenarios, given a little thought: and it’s an investment that is well-worth making because what you get is genuine learning. If you apply the AFECT criteria for psychological confidence, you will realise why you can trust the results of such a test far more than anything from a survey or focus group.
Philip Graves
That, and they remember everything about you, so when you get to their web site, they always have recommendations for you based on your last purchase. Very smart. They must have a computer system to rival the Pentagon.
Tim Van Milligan
Discover How To Get Instant Rapport With Customers
Hopefully at this point they’re only testing the revenue implications and not the usability. Online they have no idea of what you came to the website to do, versus a usability test where the tasks are assigned and the ability of a visitor to complete the test is the criterion.
I can see the role of both but in my view a split test, particularly when you have the scale of Amazon, will provide answers to usability also. If one side of the test has significantly different proportions of people buying a particular product than the other side, and if conversion is different, you can easily extrapolate where usability issues may exist. However, when people are primed to conduct a particular shopping exercise for a usability study there is no guarantee that they are in the same frame of mind as they would be when shopping outside of that artificial context: people may be more determined, for example, so as not to look foolish and ‘fail’ at the test; or they may give up more quickly because they have no real need to locate the product.
Additionally, usability isn’t static. How do you replicate the real-world effect of people trying once, giving up, returning through habit and then either succeeding or failing again? And how many iterations do you presume is reasonable before you conclude people haven’t adapted quickly enough? A live trial with a split test answers these questions with no artificial context.
@Philip Graves
I really have enjoyed your book consumer.ology, having just started a masters in consumer psychology with business I am noticing already how often I urge people to stop and think if they are really testing what they want to. Everybody knows they want their experiments to be valid in the real world, but when the criteria for judging such validity is flawed… Thanks to your book i have learnt to question “answers” alot more. As it seems amazon have done too, and will be all the more profitable for it.