Yesterday, early morning (India time) servers hosting Visual Website Optimizer stopped responding to queris. This may have slowed down your website(s). Please accept my sincere apologies for that. You cannot imagine how concerned we (Wingify team) all are for this. The servers are up at the moment. To clarify, the servers have NOT been compromised – it is just that they got slowed down.
We are transparent with customers on such issues and take responsibility for informing you on what happened, why it happened and what steps we are we taking to ensure it never happens again.
For people who understand server-admin technical terms: our server load typically ranges between 10%-15%. Somehow, yesterday during the outage it shot upto 400% which slowed down the servers. We are still investigating what exactly caused such high server activity but there are three likely causes:
1. A bug in code which resulted in repeated endless queries on database
2. A VERY, VERY highly trafficked site sending requests to the server
3. A DDoS attack on servers
We had taken proper precautions for cases like these: a) server monitoring is in place; b) daily backups are taken at an offsite secure location; c) your passwords are stored in salted hash format (and not plaintext) to ensure that nobody (not even us) can know what your password is.
All in all, we are REALLY concerned about safety of your data and smooth functioning of your websites where Visual Website Optimizer code is installed.
Just as a precaution, so that such a server outage doesn’t happen again, we are taking two immediate steps:
1. Upgrading our servers – essentially doubling the capacity
2. Upgrading the server monitoring service so that response time for such an outage can be brought down to minutes
As mentioned earlier, the servers are up now. So, if you had removed the code due to the slow response, please re-insert it. The situation is back to normal.
Once again, I personally apologise for this outage.
Thanks for understanding.
Regards,
Paras Chopra
Recently, I published an A/B split testing case study where an eCommerce store reduced bounce rate by 20%. Some of the blog readers were worried about statistical significance of the results. Their main concern was that a value of 125-150 visitors per variation is not enough to produce reliable results. This concern is a typical by-product of having superficial knowledge of statistics which powers A/B (and multivariate) testing. I’m writing this post to provide an essential primer on mathematics of testing so that you never jump to a conclusion on reliability of a test results simply on the basis of number of visitors.
What exactly goes behind A/B split testing?
Imagine your website as a black box containing balls of two colors (red and green) in unequal proportions. Every time a visitor arrives on your website he takes out a ball from that box: if it is green, he makes a purchase. If the ball is of red color, he leaves the website. This way, essentially, that black box decides the conversion rate of your website.
A key point to note here is that you cannot look inside the box to count the number of balls of different colors in order to determine true conversion rate. You can only estimate the conversion rate based on different balls you see coming out of that box. Because conversion rate is an estimate (or a guess), you always have a range for it; never a single value. For example, mathematically, the way you describe a range is:
“Based on the information I have, 95% of the times conversion rate of my website ranges from 4.5%-7%.”
As you would expect, with more number of visitors, you get to observe more number of balls. Hence, your range gets narrower and your estimate starts approaching true conversion rate.
The maths of A/B split testing
Mathematically, the conversion rate is represented by a binomial random variable, which is a fancy way of saying that it can have two possible values: conversion or non-conversion. Let’s call this variable as p. Our job is to estimate the value of p and for that we do n trials (or observe n visits to the website). After observing those n visits, we calculate how many visits resulted in a conversion. That percentage value (which we represent from 0 to 1 instead of 0% to 100%) is the conversion rate of your website.
Now imagine that you repeat this experiment multiple times. It is very likely that, due to chance, every single time you will calculate a different value of p. Having all (different) values of p, you get a range for the conversion rate (which is what we want for next step of analysis). To avoid doing repeated experiments, statistics has a neat trick in its toolbox. There is a concept called standard error, which tells how much deviation from average conversion rate (p) can be expected if this experiment is repeated multiple times. Smaller the deviation, more confident you can be about estimating true conversion rate. For a given conversion rate (p) and number of trials (n), standard error is calculated as:
Standard Error (SE) = Square root of (p * (1-p) / n)
Without going much into details, to get 95% range for conversion rate multiply the standard error value by 2 (or 1.96 to be precise). In other words, you can be sure with 95% confidence that your true conversion rate lies within this range: p % ± 2 * SE
What does it have to do with reliability of results?
In addition to calculating conversion rate of the website, we also calculate a range for its variations in an A/B split test. Because we have already established (with 95% confidence) that true conversion rate lies within that range, all we have to observe now is the overlap between conversion rate range of the website (control) and its variation. If there is no overlap, the variation is definitely better (or worse if variation has lower conversion rate) than the control. It is that simple.
As an example, suppose control conversion rate has a range of 6.5% ± 1.5% and a variation has range of 9% ± 1%. In this case, there is no overlap and you can be sure about the reliability of results.
Do you call all that math simple?
Okay, not really simple but it is definitely intuitive. To save the trouble of doing all the math by yourself, either use a tool like Visual Website Optimizer which automatically does all the number crunching for you. Or, if you are doing a test manually (such as for Adwords), use our free A/B split test significance calculator.
So, what is the take-home lesson here?
Always, always, always use an A/B split testing calculator to determine significance of results before jumping to conclusions. Sometimes you may discount significant results as non-significant solely on the basis of number of visitors (such as you may do for this case study). Sometimes you may think results are significant due to large number of visitors when in fact they are not (such as here). You really want to avoid both scenarios, don’t you?
Dustin Curtis made a wave on the Internet when he wrote a post which detailed how he increased his twitter followers by 173%. The key idea was that the phrase “You should follow me on Twitter here” did all the magic. Since then I have lost count of how many times I saw the same phrase copied verbatim on numerous other websites. (Apparently, I had also fallen for the same trap on my personal homepage).
Every time I see that copied phrase, I feel slightly sad because it illustrates the way people copy so-called best practices without testing for their specific case. There are more than 4.5 million Google results for “You should follow me on Twitter” and I am sure:
Not a single website would have tried replicating Dustin’s results by testing this new phrase against their existing text.
“You should follow me on Twitter” is not a magic potion which will increase your Twitter followers overnight. It works for Dustin Curtis because it blends well with the tone of his posts and his design. On your website, the same phrase may look terribly out of place. You cannot be requesting your blog readers to please subscribe for RSS updates and then subsequently order them to follow you on Twitter.
Of course, the underlying issue here is much broader. It is hard to avoid copying best-practices and blockbuster results others have got (especially if it is easy to copy). But it is extremely critical to test those best-practices for your case. What works for them will not necessarily work for you. Your website is unique, your business goals are unique. Copy-pasting other’s results may actually decrease website effectiveness (the good news is that you will never realize it decreased because, hey, you don’t measure website goals in the first place).
I am not to denying that you shouldn’t read successful case studies. (In face, I recently blogged how case studies is an excellent way to get test ideas) The key idea here is to:
If you were also lured into the “You should follow me on Twitter” hype, now is the best time to test it against a politer version “Please follow me on Twitter” or a a version that creates excitement “See what I am doing right now”. In fact, there are hundreds of other versions which may be better suited to your website’s theme, tone and design; proper A/B split testing is the ONLY way to find out what will work for you.
That said, you should definitely NOT follow me on Twitter.
Suppose you want to increase newsletter signups and you (rightly) decided to use A/B split testing for the task. You have all the ammo ready with you:
But you still miss one essential ingredient without which all your ammunition will turn out to be useless. That ingredient is your test plan. Hypothetically, you can test everything on your page (from footer copyright notice to your company logo). But that assumes you have infinite resources and unlimited patience. When you are doing an A/B test, you want to maximize potential returns for the time invested in creating and running the test. Since a split test takes time (2-3 weeks) to show results, choosing the right element to test and creating good variations for it becomes the most important step.
A great way to have interesting ideas for your A/B split or multivariate test is to study and read existing case studies. Case studies related to your problem (e.g. newsletter signup) and your industry (e.g. finance) tell you what was tested, which variations were created and the level of success that was seen. Great case studies go one step further and reflect on what lessons can be learnt from the test. All in all, pouring through multiple case studies is a guaranteed way to get numerous good ideas for your A/B test.
A quick note of warning: NEVER replicate the results of a case study without conducting testing on your website. What worked for them will not necessarily work for you; though what worked for them has a good chance for also working for you (but it is never guaranteed). So, best strategy is to learn from case studies, get ideas but implement your own A/B test based on the insights.
Finding relevant and interesting case studies is really hard. For your convenience, I have compiled some excellent resources on where to find case studies for A/B Split and Multivariate Tests:
As you can see there are multiple sources for interesting case studies. But finding case studies relevant to your challenge and your industry is still difficult. To help you with that task, we have created an interface to search case studies by problem type and industry type. Really happy to introduce…
This is the first iteration towards the vision of having an easy to use interface for accessing resources on A/B split testing. The toughest part of an A/B test is to come up with a good test plan and beginners can easily get de-motivated if they don’t see positive results for their first test. A/B Ideafox hopes to provide interesting testing ideas which can potentially produce successful results quickly because those ideas have already been tried elsewhere. (Search results can be a little non-specific for now but we are working hard to have refine the result quality. Leave a comment if you have ideas on on refining A/B Ideafox).
Let me know your feedback on A/B Ideafox and other case study sources by leaving a comment here. Do you think case studies really help in getting interesting ideas? What other resources (videos, screencasts, tutorials, etc.) can help for taking A/B testing to mainstream?
One of the Visual Website Optimizer’s earliest beta testers, MedaliaArt is an online art gallery specializing in Caribbean and Latin America art. For the holiday season, they put up a sale where they give 5-55% discounts on all paintings. They wanted to determine the best location on homepage to put up that message so as to optimize for bounce rate. Their sales process is long (involving phone calls, multiple visits, etc.) so they chose to measure and optimize bounce rate instead of sales conversions. As a hypothesis, providing discounts must pull in more visitors to go through multiple pages on the website exploring different paintings.
However the challenge with putting up a ‘Holiday Sale’ message is where to show it. Displaying it prominently on the homepage will make more visitors notice it but some may find it too intrusive and leave the site immediately. On the other hand, putting it at a not-so-noticeable location may have no effect at all. So, what is the best position on page to display the ‘Holiday Sale’ (or for that matter any other promotional) message?
Only a split test can answer that. (In a split test, different visitors see (randomly selected) different versions of homepage). MedaliaArt setup a split test to optimize their website for bounce rate. First, they created a couple of versions of the homepage with ‘Holiday Sale’ displayed at different locations. Of all versions, following represented two extremes:
In-your-face ‘Holiday Sale’ message displayed in big, red font prominently on the homepage.
Sidebar ‘Holiday Sale’ message in small font.
Usually, split testing tools do not track bounce rate; they rather track conversion rate (percentage of visitors doing desired action). To track bounce rate instead, MedaliaArt did a neat trick. They defined a click on any link on the homepage as conversion. Thus the conversion rate of, for example, 40% corresponded to 100-40 = 60% bounce rate.
So, which variation had a better bounce rate? Any guesses?
They started the test and after two weeks got their first batch of conclusive results.
| Message location | Visitors | Clicks (conversions) | Conversion Rate | Bounce Rate | Reduction |
| Sidebar | 145 | 35 | 24% | 76% | N.A. |
| In-your-face | 123 | 49 | 40% | 60% | -21% |
Clearly, the in-your-face, prominent promotional message has dramatically less bounce rate (60%) than the sidebar one (76%). The reduction in bounce rate of 21% is statistically significant (at 95% confidence level) so the In-your-face variation really represents a better version. The improvement in bounce rate means more interest by visitors in the paintings they are selling and potentially more sales. What they feared that a prominently displayed promotional message can backlash by irritating visitors didn’t really happen. Without split testing they could have never really known the optimal position of their promotional message. Now they know.
For the next test they do, there are a couple of suggestions for MedaliaArt (or any other eCommerce optimizing for promotional messages):
What eCommerce stores learn from this case study?
Split testing is the only way the really know what will work and what won’t. Testing is essential to check assumptions related to promotional messages, checkout process, product category ordering, buy now button, etc. Be a little adventurous and test radically different homepage designs and ideas. You can always choose to include only a small percentage of traffic and can disable non-performing variations at a click of a button. So, what’s your excuse for not using split testing for increasing sales for your eCommerce store?
This case study is also appeared in form of interview at Practical eCommerce magazine as Split Testing Can Increase Conversion Rates.
Year 2010 is going to be a blockbuster year for split testing. With web analytics going main stream in 2008 and 2009, it is now time for its little brother to be seen and heard. As a means to test hypotheses and ideas in a scientific manner, split testing should be an integral part of business strategy, just as email marketing, SEO and PPC advertising is. This is especially important as recession gets over and online businesses get back in the mode of trying out innovative (and sometimes crazy) ideas for optimizing revenue and increasing sales per customer/visitor. An excitement is building in online marketing world and split testing is a happy family member.
Why do I think year 2010 will be a year of A/B split testing? Here’s why:
Reason 1: Growing Awareness
With 1,760,000 search results on Google already, “split testing” is slowly crossing the chasm and is definitely not just a buzzword today. This transition from early enthusiasts to mainstream online businesses will accelerate in 2010 thanks to blogs dedicated to testing (Conversion Rate Experts, Widerfunnel, Widemile, Grockdotcom, etc.), regular posts from industry experts such as Avinash Kaushik, Tim Ash, etc. and efforts by A/B testing evangelists such as Patrick McKenzie and Ben Tilly.
Reason 2: Split testing is best methodology for conversion optimization
SEOMoz says 2010 is going to be a year of conversion optimization. And guess what – split testing sits at the core of conversion rate optimization process. I can hardly imagine any initiative for increasing website sales and conversions which doesn’t use A/B testing. There are literally a gazillion factors affecting website conversions and getting them all right at the same time using the designer’s or boss’ hunch is a long shot. Instead, every factor needs to be carefully tested one-at-a-time (split testing) or several at once (multivariate testing); bumping up conversion rate a little with each test. The real magic happens when all such little bumps add up to become a really huge bump in company’s bottom-line.
Reason 3: Rise of case study repositories
A usual excuse for not doing split testing is lack of ideas. People just getting started can be clueless on what elements on a website affect conversion rate. Even after they zero on to testing the right element on a page (say signup button), getting ideas for interesting variations is a big stumbling block. Fortunately, websites such as Which Test Won and ABTests.com document a lot of split testing case studies. Those case studies are an excellent starting point for what to test and what results to expect. If you want to optimize your signup rate, pick a relevant case study and test what they tested (of course, also test whatever ideas you have). You may not get results that they got (that is the whole idea of testing in the first place), but you get ideas which you can now refine. These repositories will become important in 2010 as they fatten up with even more case studies and provide essential raw materials for increasing website conversions.
Reason 4: Accessible, integrated, affordable and simple to use tools
With 15+ free A/B testing resources available on the Internet, the cost of trying it out has reduced to zero. But the technical complexity and integration hassles are a still a big turn off for many website owners. Tools such as (our very own) Visual Website Optimizer solve those pain points by being extremely simple to use and requiring no integration hassles. This essentially removes ALL barriers for trying out A/B testing. In 2010, the industry can expect a lot of innovation on usability front so that tools become even more easy to use. Split testing will also become heavily integrated with web analytics such that testing green button v/s red button becomes as routine as checking number of visitors on your website.
The world is getting ready to try out A/B testing this year, what about you?
Do you have any plans for using split testing in 2010? What is holding you back? If you want to get your hands dirty with A/B, split or multivariate testing without all its associated hassles, use invite code “vwo-blog” (without quotes) while signing up for Visual Website Optimizer. I do not guarantee that the tool will end up redefining your 2010 online strategy (though I sincerely hope that it does) but it will definitely make you realize how simple it can be to do A/B testing for increasing website sales, downloads, signups, leads or conversions. Trust me, give it a shot! It is free, after all.
With split testing, there is always a fear at back of the mind that the results you observe are not real. This fear becomes especially prominent when you see an underdog variation beating your favorite variation by a huge margin. You may start justifying that the results may be due to chance or you have too less data. So, how do you really make sure that the test results are reliable and can be trusted? Reliability here means that running the test for more duration isn’t going to change the results; whatever results you have now are for real (and not by chance).
So, how do you determine reliability of your A/B test? Hint: you don’t. You let your tool do the work for you. Visual Website Optimizer employs a statistical technique where your conversion events are treated as binomial variables. Above a certain sample size (10-15 visitors), binomial variables can be approximated to a normal distribution. Key point to note is that your conversion rate is a distribution, not a single value. That is, you always get a range (e.g. 31.3% ± 14.8%) for your conversion rate.
As you can see in the image above, a range for conversion rate is provided in the report. For statistics-geeks, this range actually represents 80% of the total area of the normal distribution. One peculiar property of the range (thanks to a concept called standard error) is that initially it is very wide but as more data gets collected, with time, it becomes narrower and narrower. Moreover, with time, the estimate of true conversion rate becomes preciser. For example, if on day 2 of your split test you observe that your conversion rate for a variation is 50% ± 25%, after a week you may observe that it has changed to 40% ± 8% (note that conversion rate has changed and the range has become narrower).
The way to be sure that your results are reliable is to compare conversion rate ranges of different variations and see if there is NO overlap between them. You can visualize this overlap in the chart above. Observe that there is little or no overlap between Control and “free download” variations. So you can be pretty sure that this result is reliable and “free download” indeed works better than the control (which in this case was a simple “download”). This overlap (in distributions) can also be calculated numerically and Visual Website Optimizer calculates it as “Chance to Beat Original” metric. If that value is >95%, you can be pretty confident that the variation will be better than the control.
If you don’t really trust the statistics (by the way, there is no way you shouldn’t), you can still be confident about the test results by employing a neat trick. The idea is to have two identical variations (usually of the control) and see if there is any difference in their conversion rates. Of course, a minor difference will be there (due to randomness) but if you see a large difference between conversion rates of identical variations, you shouldn’t trust the results.
In upcoming posts, I’m going to describe the maths behind split testing, so stay tuned.
