Posted in A/B Split Testing on August 21st, 2013
It all seems so easy. You have two versions of your website. Version A is your original page. And in version B, you’ve probably tweaked button color or something like that. All you need to do now is wait…and see which version wins. Simple!
At Visual Website Optimizer, this is exactly what we set out to do – make A/B tests very easy to setup and run. No need to mess with the code, no chasing after the designers to get small tweaks to implement the test. But there’s only so much that your A/B testing software can do for you.
To get the best returns from your AB testing tool, it’s important that you know how to exploit it for maximum benefit. Listed below are the indications that you are not making the optimum use of your AB testing tool and how you can use it better:
1) You Are Not Segmenting Your Test Visitors
Visitor behavior on your website can vary due to many reasons. They might be from different income groups, geographical location, marketing channels from which they arrived on your website (PPC, organic, social media, and others). This means, their need or intention of visiting your website can be different. As a result, the way they interact with your website will also vary.
When you study the behavior of these individual segments, it can give you highly actionable data to improve your conversions. Avinash Kaushik, mentions in his post, The Choice is Stark: Segment or Die:
The best ideas for taking action come from the process of segmentation. Put simply it is taking the entirety of the data on your website and breaking it down into meaningful chunks. Knowing your overall conversion rate is 2%, useless. Knowing the conversion rates of your main acquisition channels are: Paid Search: 9%, Direct: 6%, Affiliate: 0.58%, priceless.
Apart from missing out on the actionable data, sometimes not segmenting your tests can give you really skewed results as well. For example, one of our agency customers set up a test for their client, Jelly Telly, which is a Christian videos streaming website. The site follows a membership-business model, much like Netflix. And signing up for their membership provides unlimited access to their videos and online games for kids. They tested if navigation elements on their homepage are a distraction and increased their conversions by 105%.
Before starting the test, they saw that a large number of visitors to the website were already members. This is the reason why they segmented out returning visitors from the test for more accurate conversion results.
Many people hesitate from segmentation thinking that it’s difficult to execute. But in most cases, it’s usually not. In Visual Website Optimizer, the most commonly used segments are available for you to use. All you really need to do is decide which segment(s) are most appropriate for your website/test. You can even set multiple segments simultaneously if you want. It hardly takes a minute.
If you’re thinking about how to figure out which segments are more appropriate for your business type, watch this awesome video by Avinash Kaushik that discusses five actionable segmentation ideas you can try.
2) You Love Random Test Ideas
It’s amazing to see 5-10% win or even more from something as small as adding or replacing one word in the headline. It may seem exciting initially but tests like these can set you up for random “test whatever” approach.
This can leave you with a lot of inconclusive data and no idea about the actual obstacles that are truly stopping your customers from taking those final steps to conversions. And this is mainly because you’re too focused on micro-conversions that are not based on any quantifiable learning.
Random tests are more likely to leave your team frustrated with too many negative test results and no payoffs for their efforts. This can drain the entire momentum of your testing program. Often people who do not put much thought into their testing ideas and strategies are are the ones that are seen saying statements like, “AB testing doesn’t work.”
Testing is about improving conversions of a website. But what many people fail to realize is that it’s also about understanding your customers better. Only when you know what your customers want and how they perceive your brand, you’ll be able to make the real difference to your conversions. When you know what your customers expect from you and you give it to them, your conversions increase.
There are several ways you can try to understand visitor behavior on your site:
- You can use heatmaps and clickmaps data to see if they are scrolling down a page, where they are clicking the most, and other such stuff.
- You can collect customer feedback by conducting surveys on your website. Michael Aagaard of Content Verve asks his readers which eBook topic would interest them the most.
- To send full-length surveys to your customers, you can use Survey Monkey. Read this post to know how you can create good surveys that do not overwhelm visitors and how you can get actionable insights from them.
- Usability testing can help you figure out the major roadblocks in your website. In the video given below, you can see how the usability guru, Steve Krug, conducts a usability test.
Proper usability tests that are conducted in a controlled environment can be very expensive. But if you are on a limited budget, you can use remote usability testing in your VWO app or submit your questions on Qualaroo. You can read my post to know more about getting actionable insights and accurate results from usability testing.
Findings from these research methods or insights drawn from your previous tests, along with reading about user experience or conversion optimization, can then be used to come up with good hypotheses. And you can then A/B test them with your live visitors to see how they perform.
Understanding your customers and having a well-researched hypothesis can get you great results.
3) You Don’t Care About the Sample Size
The moment your test hits that 95% mark and declares your variation a winner, you jump with joy. Is that how it goes? Did you see that there are only 200 visitors that have been a part of your test until now? Don’t you think this sample size is a bit too small to draw a conclusion about the winning variation when your site receives 10,000 visitors every day?
When you make your winning variation permanent on your website with insufficient sample size, chances are that the actual long-term conversion rate from your website will not match the conversion results in your AB testing tool (even if your result in the software was statistically significant).
To avoid making such bad decisions, you must figure out the number of visitors you will need before you begin a test. You can use our free split testing duration calculator to find the time period for which you should run the test (according to the average visitor count of your site) to get reliable results.
4) You’re Not In for the Bigger Wins
You can do all the copy tweaks and color changes that you want. Go ahead and play around with your call-to-action buttons too. But sooner or later you will reach the local maxima, which means that your current design will reach its optimum conversion stage. And after this, almost any tweaks that are made by you will only give you negative test results.
So, if bigger wins is what you want, you will need to have a broader view of your conversion optimization strategy. From your business model to your pricing strategies, there’s so much you can test. Moz got a radical makeover, which increased their sales by 52% and added $1 million to their revenue.
As long as you support your hypotheses with valid data and research, you shouldn’t hesitate in trying bigger changes. Two things that you must keep in mind while coming up with bigger changes are:
- Figure out how to get your visitors closer to the must-have experience or feature of your product or service in the minimum steps/clicks possible.
- Try to minimize friction by making changes in your conversion funnel or by making changes in user flows.
There’s a lot that you can achieve with your current visitors if you tap the power of A/B testing the right way. You might use the best AB testing software in the market but if you take any of the above points for granted, you’re missing out lucrative opportunities for yourself.
Know all that you need to get started:
- What is A/B Testing?
- Is it compatible with SEO?
- How to start your first A/B test?