Metrics can be a touchy subject in design. When I say things like, “Designers should embrace A/B testing” or “Metrics can improve design,” I often hear concerns.
Many designers tell me they feel that metrics displace creativity or create a paint-by-numbers scenario. They don’t want their training and intuition to be overruled by what a chart says a link color should be.
These are valid concerns, if your company thinks it can replace design with metrics. But if you use them correctly, metrics can vastly improve design and make you an even better designer.
First, when I talk about metrics, I’m talking about making use of a couple of very specific tools:
Shopping Cart Abandonment Infographic by KISSmetrics
A/B and multivariate testing involve looking at how changes affect key metrics such as revenue and retention. When you run an A/B test on a change, you’re comparing user behavior with the new design to user behavior with the old design in order to answer the rather important question of “Which design caused the user to do more of what I wanted and less of what I didn’t want?”
By incorporating analytics and A/B testing into their process, designers can not only improve the business outcomes of their redesign projects, but also become better at design.
Metrics are fabulous for making these decisions. For example, analytics can show exactly how much of a drop-off your shopping cart is getting. It can show you precisely how many visitors aren’t converting into signed-in users. It can pinpoint how many people bounce right off of product pages.
Metrics tell you where your biggest business problems really are, so that you can use design to fix them.
Of course, metrics can’t tell you how to actually fix the problems. Your job as a designer is to learn why users might be having problems in certain areas of your product and come up with brilliant solutions.
(Image: Adriano Gasparri)
Answers to questions like these can have an enormous impact on the way you design. After all, depending on whether users have no more than a few friends or they tend to have more than a hundred friends, the interaction for selecting from a list can be quite different.
Metrics and analytics free the designer from having to guess the answers to questions like these.
Again, knowing the answers doesn’t tell you how to design the product, but it does give you a lot more insight into the real problems you’re trying to solve.
Or if you’re trying to decide what sort of widget to use to access some data, remember that the right solution often depends on how much data there will be, and that question is entirely answerable as well.
Design changes — like code changes, marketing changes and customer service changes — should eventually make a company more money, because that’s how the company will stay in business and pay the designer’s salary.
With metrics, you can learn exactly what sort of an effect your changes will have on the numbers you care about.
Let’s say you’ve been obsessing over the navigation of your website. You feel it’s confusing for users and is frustrating them and causing them to leave. So, you revamp the navigation and run an A/B test so that half of users see the old version and half see the new one.
After a few thousand people have seen both versions, you have a very good idea whether changing the navigation has resulted in good things like increased revenue, time on site, and the number of users who would recommend you to a friend.
Of course, not every change you make will have a direct and significant impact on revenue. But knowing that any major change you make is having a measurable influence on the things you care about is very nice.
As a bonus, once you’ve made a few demonstrable improvements to revenue, asking for a raise or for more design resources becomes a lot easier.
Obviously, you can make a few changes that you know will negatively affect certain metrics because they’re part of a larger strategy. But at some point you need to put a stake in the ground and say, “If we make the following design changes, then important metrics such as revenue and retention will improve.”
If you test all of your design changes against the status quo, you should eventually see those key metrics go up as a result of your work.
Knowing that all of your changes will be A/B tested gives you a tremendous amount of freedom to try new and potentially dangerous things. If they pay off, you’ll find out immediately. If they cause problems, you can identify and fix them quickly.
You no longer need to worry that some change you’ve made will mysteriously ruin everything. And if you weren’t worried about that before, you probably should have been, because even very small design and text changes can have an enormous impact on user behavior.
Which blue is better? (Image: visualpanic)
In fact, metrics are terrible for a lot of things that design does well. For example, metrics can’t tell you how exactly to improve a design. They can only tell you whether your design is better or worse than another. They can tell you exactly what users are doing, but they can’t tell you why they’re doing it or how to make them stop.
Metrics can be an incredibly powerful tool, but they don’t replace design or make it irrelevant. In the end, the designers are still the ones making the decisions. They’re just making them with better information.
Many designers tell me they feel that metrics displace creativity or create a paint-by-numbers scenario. They don’t want their training and intuition to be overruled by what a chart says a link color should be.
These are valid concerns, if your company thinks it can replace design with metrics. But if you use them correctly, metrics can vastly improve design and make you an even better designer.
First, when I talk about metrics, I’m talking about making use of a couple of very specific tools:
- User analytics,
- A/B or multivariate testing.
Shopping Cart Abandonment Infographic by KISSmetrics
A/B and multivariate testing involve looking at how changes affect key metrics such as revenue and retention. When you run an A/B test on a change, you’re comparing user behavior with the new design to user behavior with the old design in order to answer the rather important question of “Which design caused the user to do more of what I wanted and less of what I didn’t want?”
By incorporating analytics and A/B testing into their process, designers can not only improve the business outcomes of their redesign projects, but also become better at design.
[Editor's note: A must-have for professional Web designers and developers: The Printed Smashing Books Bundle is full of practical insight for your daily work. Get the bundle right away!]
Metrics Tell You Where Design Is Needed Most
Unless your resources are unlimited, prioritizing design changes can be tricky. For example, if you’re working on an e-commerce website, who’s to say whether optimizing the sign-up flow is more important than revamping the check-out flow or rebuilding a product page or simplifying the navigation? Any of these could be pretty good ideas.Metrics are fabulous for making these decisions. For example, analytics can show exactly how much of a drop-off your shopping cart is getting. It can show you precisely how many visitors aren’t converting into signed-in users. It can pinpoint how many people bounce right off of product pages.
Metrics tell you where your biggest business problems really are, so that you can use design to fix them.
Of course, metrics can’t tell you how to actually fix the problems. Your job as a designer is to learn why users might be having problems in certain areas of your product and come up with brilliant solutions.
How Should You Use This?
When deciding on your highest design priority, examine your product’s analytics to identify where your biggest problems are.Metrics Help You Track Real User Behavior
How many times have you been designing and thought something like, “I wonder how many products people have in their shopping carts on average when they drop out?” Or, “What is the average number of friends a user has in such-and-such a social network?”
(Image: Adriano Gasparri)
Answers to questions like these can have an enormous impact on the way you design. After all, depending on whether users have no more than a few friends or they tend to have more than a hundred friends, the interaction for selecting from a list can be quite different.
Metrics and analytics free the designer from having to guess the answers to questions like these.
Again, knowing the answers doesn’t tell you how to design the product, but it does give you a lot more insight into the real problems you’re trying to solve.
How Should You Use This?
The next time you’re debating with someone about where users are dropping out of a flow or how many users are watching a video tutorial, remember that these questions are easily answered by metrics.Or if you’re trying to decide what sort of widget to use to access some data, remember that the right solution often depends on how much data there will be, and that question is entirely answerable as well.
Metrics Tell You Which Changes Are Most Effective
Let’s be clear. Your job as a designer is to improve the customer experience in some measurable way, right? If you do a complete redesign, and every designer you know loves it, and it wins awards from experts, but every single one of your users hates it and leaves and stops giving you money, then that’s a failure for the company.Design changes — like code changes, marketing changes and customer service changes — should eventually make a company more money, because that’s how the company will stay in business and pay the designer’s salary.
With metrics, you can learn exactly what sort of an effect your changes will have on the numbers you care about.
Let’s say you’ve been obsessing over the navigation of your website. You feel it’s confusing for users and is frustrating them and causing them to leave. So, you revamp the navigation and run an A/B test so that half of users see the old version and half see the new one.
After a few thousand people have seen both versions, you have a very good idea whether changing the navigation has resulted in good things like increased revenue, time on site, and the number of users who would recommend you to a friend.
Of course, not every change you make will have a direct and significant impact on revenue. But knowing that any major change you make is having a measurable influence on the things you care about is very nice.
As a bonus, once you’ve made a few demonstrable improvements to revenue, asking for a raise or for more design resources becomes a lot easier.
How Should You Use This?
Get into the habit of A/B testing your design changes.Obviously, you can make a few changes that you know will negatively affect certain metrics because they’re part of a larger strategy. But at some point you need to put a stake in the ground and say, “If we make the following design changes, then important metrics such as revenue and retention will improve.”
If you test all of your design changes against the status quo, you should eventually see those key metrics go up as a result of your work.
Metrics Give You The Freedom To Take Design Risks
Big design changes can involve big risks. After all, if what you’re proposing has the potential to vastly increase revenue, then it also has the potential to vastly decrease revenue.Knowing that all of your changes will be A/B tested gives you a tremendous amount of freedom to try new and potentially dangerous things. If they pay off, you’ll find out immediately. If they cause problems, you can identify and fix them quickly.
You no longer need to worry that some change you’ve made will mysteriously ruin everything. And if you weren’t worried about that before, you probably should have been, because even very small design and text changes can have an enormous impact on user behavior.
How Should You Use This?
Be bold! The next time you’re inclined not to make an important change because you’re concerned it might have a negative impact, go ahead and try it. Roll the change out to 10% of your users, and get real data on whether it helps or hurts the product.What Metrics Won’t Do
A lot of the push-back I get on measuring design is based on a fundamental misunderstanding of A/B testing and analytics. Some designers hear about Google testing dozens of shades of blue, and they feel that design shouldn’t be reduced to that level. After all, design is more than about picking a shade of blue.
Which blue is better? (Image: visualpanic)
In fact, metrics are terrible for a lot of things that design does well. For example, metrics can’t tell you how exactly to improve a design. They can only tell you whether your design is better or worse than another. They can tell you exactly what users are doing, but they can’t tell you why they’re doing it or how to make them stop.
Metrics can be an incredibly powerful tool, but they don’t replace design or make it irrelevant. In the end, the designers are still the ones making the decisions. They’re just making them with better information.
No comments:
Post a Comment