Red Gate software runs an annual challenge where they buy a small software company for a million dollars. They list a number of requirements that the software company must fulfill. One of the requirements that stood out was about conversion rate. They said:
If you’re selling your product then it must have at least a 10% conversion rate.
This requirement actually made me say “Wow, that’s insane”. Let me elaborate why.
Conversion rate is percentage of visitors who actually bought something on your website. Let’s imagine there are two websites: one sells product X with 5% conversion rate and the other one sells product Y with 10% conversion rate. Now, here is a million dollar question:
Is 10% conversion rate of product Y better than 5% conversion rate of product X?
It’s foolish to even begin answering above question without considering following factors:
In nutshell, conversion rate by itself doesn’t tell much (unless you have extra information like traffic, sales price, lifetime value, traffic mix, etc.) So a website with 1% conversion rate may not necessarily be worse as compared to a website with 10% conversion rate. Conversion rate in isolation is a useless metric.
Conversion rates are not entirely useless. In fact, they are very useful when seen on a temporal scale. In other words:
If your conversion rate is 5% today, aim should be to increase it to 7% (using A/B testing, etc,) or at least not let it fall to 3%.
So, comparing conversion rate over time makes a lot of sense (but for the same website). Unless you have a lot of other information about your competitors, you should NOT obsess over comparing your conversion rate to their conversion rate and whether it is lower/higher. Instead, you should obsess on how you can increase your conversion rate (since that’s one of the easiest things to make your bank balance fatter).
Note: if you go through our library of A/B testing case studies, you will note that we always talk about increase in conversion rate and not conversion rate per se.
I just published a guest post on Smashing Magazine titled Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates.
Essentially, there are five steps to increasing conversion rate:
If these steps sound complicated to you, I recommend you to read the extensive tutorial which has numerous examples.
My article explains multivariate testing by means of a case study where I tested following variations on a software download page (notice color and text changes):
Can you guess which variation produced maximum downloads? Well, the end result of this test was that #10 combination (in the screenshot above, one with ‘Download for Free’ in red) had 60% improvement in conversion rate. That’s the power of multivariate testing.
Read the full case study and tutorial: Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates. I hope you like it!
When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.
Background
Soocial is an online address book that helps you keep your phone, computer and online services contacts sane. The app syncs, merges and provides backups so that you are never stranded without the contact details you need. It works with over 500 phone models, webmail (like Gmail, Hotmail and Yahoo) and your Mac and Outlook. Soocial chose to use Visual Website Optimizer for their testing needs. Their first test was to increase the click-through rate on the homepage to the signup form.
What was tested
Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.
Results
As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.
Control: 14.5% conversion rate
Variation: 18.6% conversion rate
The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.
Why “It’s Free” Worked
When we asked Soocial why they thought the winning variation worked, this is what they had to say:
Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.
So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:
To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.
For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.
How valuable was Visual Website Optimizer for the A/B Test?
Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:
Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.
Key Takeaways
Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!
Recently, I published an A/B split testing case study where an eCommerce store reduced bounce rate by 20%. Some of the blog readers were worried about statistical significance of the results. Their main concern was that a value of 125-150 visitors per variation is not enough to produce reliable results. This concern is a typical by-product of having superficial knowledge of statistics which powers A/B (and multivariate) testing. I’m writing this post to provide an essential primer on mathematics of testing so that you never jump to a conclusion on reliability of a test results simply on the basis of number of visitors.
What exactly goes behind A/B split testing?
Imagine your website as a black box containing balls of two colors (red and green) in unequal proportions. Every time a visitor arrives on your website he takes out a ball from that box: if it is green, he makes a purchase. If the ball is of red color, he leaves the website. This way, essentially, that black box decides the conversion rate of your website.
A key point to note here is that you cannot look inside the box to count the number of balls of different colors in order to determine true conversion rate. You can only estimate the conversion rate based on different balls you see coming out of that box. Because conversion rate is an estimate (or a guess), you always have a range for it; never a single value. For example, mathematically, the way you describe a range is:
“Based on the information I have, 95% of the times conversion rate of my website ranges from 4.5%-7%.”
As you would expect, with more number of visitors, you get to observe more number of balls. Hence, your range gets narrower and your estimate starts approaching true conversion rate.
The maths of A/B split testing
Mathematically, the conversion rate is represented by a binomial random variable, which is a fancy way of saying that it can have two possible values: conversion or non-conversion. Let’s call this variable as p. Our job is to estimate the value of p and for that we do n trials (or observe n visits to the website). After observing those n visits, we calculate how many visits resulted in a conversion. That percentage value (which we represent from 0 to 1 instead of 0% to 100%) is the conversion rate of your website.
Now imagine that you repeat this experiment multiple times. It is very likely that, due to chance, every single time you will calculate a different value of p. Having all (different) values of p, you get a range for the conversion rate (which is what we want for next step of analysis). To avoid doing repeated experiments, statistics has a neat trick in its toolbox. There is a concept called standard error, which tells how much deviation from average conversion rate (p) can be expected if this experiment is repeated multiple times. Smaller the deviation, more confident you can be about estimating true conversion rate. For a given conversion rate (p) and number of trials (n), standard error is calculated as:
Standard Error (SE) = Square root of (p * (1-p) / n)
Without going much into details, to get 95% range for conversion rate multiply the standard error value by 2 (or 1.96 to be precise). In other words, you can be sure with 95% confidence that your true conversion rate lies within this range: p % ± 2 * SE
What does it have to do with reliability of results?
In addition to calculating conversion rate of the website, we also calculate a range for its variations in an A/B split test. Because we have already established (with 95% confidence) that true conversion rate lies within that range, all we have to observe now is the overlap between conversion rate range of the website (control) and its variation. If there is no overlap, the variation is definitely better (or worse if variation has lower conversion rate) than the control. It is that simple.
As an example, suppose control conversion rate has a range of 6.5% ± 1.5% and a variation has range of 9% ± 1%. In this case, there is no overlap and you can be sure about the reliability of results.
Do you call all that math simple?
Okay, not really simple but it is definitely intuitive. To save the trouble of doing all the math by yourself, either use a tool like Visual Website Optimizer which automatically does all the number crunching for you. Or, if you are doing a test manually (such as for Adwords), use our free A/B split test significance calculator.
So, what is the take-home lesson here?
Always, always, always use an A/B split testing calculator to determine significance of results before jumping to conclusions. Sometimes you may discount significant results as non-significant solely on the basis of number of visitors (such as you may do for this case study). Sometimes you may think results are significant due to large number of visitors when in fact they are not (such as here). You really want to avoid both scenarios, don’t you?
