The statistics of A/B testing results can be confusing unless you know exact formulas. Earlier, we had published articles related to mathematics of A/B testing and also have a free A/B testing calculator on the site to see if your results are significant or not. The articles provides an introduction and calculator simply provides an interface; the real formulas used for calculate statistical significance of split testing results are still missing.
Excel sheet with A/B testing formulas
So, we have come up with a FREE spreadsheet which details how exactly the significance is calculated. You just need to provide e thnumber of visitors and conversions for control and variations. The spreadsheet will automatically calculate for you significance, p-value, z-value and other relevant metrics for any kind of split testing (including Adwords). Of course, you can see the relevant formulas in the spreadsheet. Click the screenshot below to download the calculator (spreadsheet):
Click here to download A/B testing significance calculator (excel sheet)
Please feel free to share the file with your friends and colleagues or post it on your blog / twitter.
PS: By the way, if you want to do quick calculations, we have a version of this calculator hosted on Google Docs (please make a copy of the Google Doc sheet into your own account before you make any changes to it).
Traffic coming to a specific blog post rarely goes on to read your other blog posts. A visit to homepage or product page from a blog reader is even rarer. No wonder, blogs have unusually high bounce rates. The Visual Website Optimizer blog gets a ton of traffic, but only a trickle of it ultimately converts into signing up for the free 30 day trial.
Our original blog design is shown below; note that we had a huge ‘I love Split Testing’ heading towards the top of the page. Even though that’s the name of the blog and ideally it should be retained, our hypothesis was that it may actually be acting as an attention barrier. Note that it competes for attention with the top navigation bar which has product logo and other links related to Visual Website Optimizer.
Original blog design (with header) -
Enter ‘Anti A/B testing‘ – removing the top header
We decided to do a simple anti A/B test to increase signups from the visitors (like you) who have come to read articles on this blog. Now, wait, you may ask what exactly an anti A/B test is. Well, it’s nothing but a term we coined to describe a test where you test presence of an element v/s absence of it. In a typical A/B test, you create variations of an element (say headline, call-to-action buttons, images, etc.). However, in an anti A/B test you remove the element altogether and then see the impact on conversions. In our anti-A/B test, we removed the blog header completely. See below how the blog now looks like without the header:
Removed header from blog: +60% increase in signups
Test Results
Guess what? The variation which did not have the blog header increased signups by 60%. And not just the signups, just about every conversion goal that we tracked showed a dramatic improvement. Have a look at the results below:
| Conversion Goal | Increase in conversion rate |
| Signups for free trial | +60% |
| Engagement rate (inverse of bounce rate) | +20% |
| Visit to signup page | +30% |
| Visit to homepage | +13% |
| Visit to blog main page | +78% |
As you can see, even visits to the main blog page increased by 78% for the variation that did not have the blog header. The overall positive results obtained from the test re-enforced our belief that even small changes on the pages can have dramatic results. (See other A/B and multivariate test case studies for more examples)
Which anti A/B test will you do?
So, next time you think about optimizing one of your pages, try to think about which element on page you can afford to remove. Perhaps it’s the large product screenshot or video that’s hurting the conversions? Or, perhaps it is the trust / secure logo that’s decreasing the conversions?
I recently got interviewed on Unbounce blog as a conversion hero. In the interview, I shared few rules-of-thumb related to A/B and multivariate testing which you may find helpful. I developed these heuristics while observing and advising hundreds of tests created by Visual Website Optimizer users. So, in this post I will paraphrase and expand on some of the things I shared in the interview.
A/B or Multivariate, which test methodology to choose?
Three main criteria will help you in choosing the right methodology between A/B testing or Multivariate testing:
Quoting from the interview, here is an elaboration on these three factors:
The eligibility criteria for each method is traffic of course. You should not attempt to do MVT if you don’t have enough traffic on the site. But assuming traffic isn’t a constraint, MVT works best when you are hyper-optimizing. That is, when your aim is to squeeze the last drop of conversion rate juice from your existing design. On the other hand, A/B testing should be used if you want to test completely different designs and ideas. Ideally, an organization should do lots of MVT tests followed by a few large A/B tests.
MVT typically requires less design resources as compared to large scale A/B test changes. Moreover, as I said, if the objective is to optimizing existing design MVT (or single element change) is way to go. But if you want to do radical changes on the page (say layout change, theme change, etc.) you will go with A/B testing.
Best methodology to start with?
Undoubtedly, if you are just getting started with testing and conversion rate optimization, you should go ahead with a simple A/B test. Multivariate testing is a complex methodology and it is easy to draw erroneous conclusions. From the interview:
For the starters, I always recommend to start with small-step changes in order to truly appreciate the value of testing. Ideally, they should pick a sweet spot on their page (ideal candidates: call-to-action, headline and image) and optimize that by a simple A/B test. Only once they get the hang of the whole process, they should attempt MVT or large-scale A/B test.
What to test and what not to?
Of course, what you test on a page depends on the specific site and objectives of the test. But if you are looking for some rules-of-thumb on what are the most common elements on a page that can be tested, here are they:
As far as what not to test is concerned, it is best to avoid testing:
What kind of surprises can you expect while doing A/B testing?
Technically no winning variation in a test should be seen as a surprise because there had to be a specific hypothesis on why you included it in the test. Nevertheless sometimes one finds that the test results are contrary to what was expected. That is, a variation won hands-down when one such expected it to lose significantly (or vice versa). Here are a few (real-world) examples of such surprises:
A recent test was very surprising – in this test it was found out that removing a secure icon from the page actually increased conversions by 400%. Another surprising result was that by simply adding a human photo on a homepage, conversion rate can be potentially doubled.
One of the test results on our homepage goes really against the standard advice of having a ‘Signup’ button prominently features on homepage. We found that a ‘Signup’ button actually decreased eventual sign-up’s and ‘Watch a short video’ worked much better because after watching the video, visitors were sure of what they are signing up for. (We had a ‘Signup’ button on the video page, by the way).
I hope you liked the interview snippets. If you want to read the full interview, head over to Conversion Heroes Part 3: Split Testing – An Interview with Paras Chopra.
Good news, Visual Website Optimizer was covered in TechCrunch a few days back! For those who aren’t aware about this mighty blog, it is considered to be one of the most influential technology blogs on the planet. Getting covered on TechCrunch is a (ego) milestone for many web startups and it was surely one of the events that we keenly looked forward to. This blog post summarizes our perspective on getting covered on a major blog and what effect did it really have on us.
How we managed to get covered by TechCrunch?
Love it or hate it, all major technology blogs (TechCrunch, Mashable, ReadWriteWeb, etc.) get more than a hundred pitches from startups daily. Even if your startup is truly spectacular, getting them to notice you could be a big deal. With so many pitches to evaluate, the blogs have justifiably come to evolve some heuristics. For example, if your pitch comes through a trusted source (say a VC or a someone journalist personally knows), it will be at least taken seriously (and not piled up along the 100s of daily pitches these blogs get). Better still, if you have a good pedigree (founded a successful startup before?) or belong to an incubator program (such as Y-combinator or Techstars), it is pretty likely that your startup will be covered on such major blogs.
So, does that mean it game over for self-funded startups who have no “special connections” (such as your very own: Visual Website Optimizer)? No, we are a proof that the TechCrunch club is not hyper-exclusive and if your startup has a good product or service, they will eventually cover you. Yes, it took us more than two months of follow ups and reminders but our persistence paid off eventually. The key word while pitching to major blogs is persistence. Realize that they get truck-loads of pitches daily and it is natural that they can’t write about each one of them. But by being persistent in asking them to at least evaluate your pitch, you differentiate yourself by showing that you are not mass-mailing your pitch and are really serious about getting covered on that particular blog!
Of course, you don’t want to annoy the journalists by not respecting their time. Sending a gentle reminder or fresh updates about your startup (such as new features, traction, social proof, new customers) every two weeks or so should be enough.
The effect of getting covered by TechCrunch
Here is a screenshot from our Analytics program:
As you can see, TechCrunch sent about 3100 visitors. To be honest, this is less than what we had expected. Probably this was because we were covered on Friday as traffic dropped to 360 on Saturday and then to 95 on Sunday. Hence, it’s best to pray that you get covered on a weekday (it is said that Tuesday is the best).
A great thing about the TechCrunch traffic is that it’s very engaging. In every engagement metric (watching video, time on site, bounce rate, pages per session), that traffic was 20-30% better than our average site traffic.
However, this audience (even though was more engaged) had converted (free trial signups) worse than the average traffic. In fact, the drop in conversion rate for TechCrunch traffic was 34% less than normal traffic. One plausible hypothesis is that the TechCrunch blog readers come across new startups on a daily basis. So, while they are curious about what this new shiny thing is about, very less actually try it out. (That said, even though conversion rate this traffic was less, remember that all these conversions are a bonus for us as they are in addition to number of signups we would normally get through our regular site traffic.)
Indirect effects
Though we don’t have exact numbers here, we could sense a definite explosion of interest last week. We got contacted by a few VCs/angels and also there were a couple of enquiries for partnerships. We also believe that there was a definitive bump in our SEO rankings for key terms: a/b testing, multivariate testing and split testing. The number of bookmarks on Delicious also surpassed 1000 (in fact, 1123 bookmarks as of today). A great side-effect of this was that many of our existing customers noticed the TechCrunch post and it reinforced their confidence in the product and the team. (Many of them personally congratulated us, which surely felt good!)
The TechCrunch effect is short-lived but worth it
TechCrunch covers a lot of startup/technology related news. Within a week of getting covered, we are already on page 10 of the blog so the impact of the post is short-lived. So, what we have learned from all this? Within a week, things will get back to normal. The world doesn’t really change if your startup gets covered or not, but the immediate exposure and validity that TechCrunch brings is really worth the effort of asking them to write about your startup.
Good luck to your startup if you pitch!
Last week, we ran a contest (with $3000 worth prize) where we asked you to predict which variation in an A/B test produced 400% increase in conversions. We received over 50 entries and (ironically) the split of answers was roughly 50/50. This shows just how hard it is to predict A/B test winner in advance. Only real data tells the truth!
For this case, now you have the answer: the variation that DID NOT have security badge actually increased conversions by 400%. This is in stark contrast to one of our earlier A/B testing case studies where a variation which had security badge increased conversions by 32% (over the control which did not have such badge). This shows that every site is unique and best practices shouldn’t be adopted without doing any testing to prove that they would work for your site too.
Out of all the correct entries, we’ve selected one entry randomly to win this contest:
The $3000 worth prize (twelve month Visual Website Optimizer $249/mo subscription) goes to: Subhash Surampudi [link to his comment]. Congratulations to the winner!
Case Study
ICoupon Blog is a coupon website that has been around for about 5 months and provides coupons and rebates for different services and products. They decided to use Visual Website Optimizer to increase click-throughs on these coupons.
What they tested was whether having a large ‘Secure’ icon in the sidebar would help with our conversion rate. They thought the icon would help because it would push down other (distracting) text in the sidebar. [Note to ICoupon: a great follow up test would be to further test this hypothesis by having a variation where sidebar text is completely removed]
Here are the variations they tested:
– Version A: with Security badge in sidebar –
– Version B: without Secure badge in sidebar –
Reasons why this particular variation was tested
In the words of Bradley Spencer (from ICoupon):
Actually I thought the ‘Secure’ icon would win hands down. It was my idea and I wanted to prove to my partners that it was a good idea. I was really surprised to learn that it didn’t help at all.
They also had some user testing done and everyone seemed to notice the ‘Secure’ button a lot. It was one of the most prominent parts of the page. So they think removing it helped focus the user on they we really wanted them to push.
A/B test results
They found that without the secure icon had over 400% improvement on conversions as compared to having the image. [Note: results are statistically significant]
Strangely, removing the icon dropped site engagement by about 17%. However, with their business model a ‘bounce’ is actually a good thing, so site engagement isn’t as large of a factor.
In nutshell, in this particular case, it looks like the secure icon distracted the visitor from the main focus of the page.
Lessons
As Bradley sums up the lessons for others:
Make each page designed to get the user to do one thing, and try to focus all of their attention on that one thing.
He also adds a word about the tool they used for the test:
Visual Website Optimizer was invaluable. We would have kept wasting user’s attention with that icon without VWO.
Today we’ve released a change in Visual Website Optimizer code snippet that will make the process of integrating VWO to your website a lot easier! As you will see, the change is a significant improvement over the previous code snippet and we recommend you to update it on your website as soon as possible to enjoy all its benefits (of course, it is backward compatible; So your existing VWO code will also work — but the new one is much faster).
The new code is cleaner and much easier to implement
The old snippet required you to insert code at two places: one in header and one in footer. The new code requires only one small snippet to be inserted into the website header. Yes, that’s it! No more you are required to add separate code snippets in header and footer.
Now, all the code required to run VWO, create and run unlimited number of A/B and multivariate tests, measure conversion rates, track heatmaps and clickmaps, etc. is contained in one snippet at a single location. This is what we call as single-tag integration! See, the screenshot below on which code snippet you need to add to integrate VWO with your website:
The new code is faster
The old code downloaded the static JavaScript file (required to run VWO) on all pages irrespective of whether a test is running on the page or not. It meant that the static file was downloaded for all your site visitors even if you are not running any test or test is only running on certain parts of your website.
The new code downloads the static JavaScript file only if and when it is needed. That is, the file is downloaded if a test is to be run on a particular page. Otherwise, it is simply skipped. What this means is a much faster experience for visitors to your website who aren’t included in the test.
We strongly recommend you to update to the new code! You can find the updated code in your test reports (under section Add code to website section)
UPDATE: The results of this A/B test and the winner of the contest have been announced! Congrats to Subhash Surampudi for winning the contest!
In the past we have published numerous A/B testing case studies which showed how even little changes could lead to dramatic improvements in sales, downloads, and and conversions. Today, we have a fresh case study where the user was able to achieve a whopping 400% increase in conversions. Yes, read that once again: a mind-boggling 400% increase in conversions (and that too statistically significant) with just a single change. We decided that from now onwards before publishing the full case study on the blog, we will run a teaser contest.
Challenge: Guess which version is the actual winner in this A/B test.
Prize: Out of all the correct entries, one lucky person will win a full year of $249/month Visual Website Optimizer subscription!! Twelve months of our Large Agency plan is total worth $3000 (12*249).
How to enter?: Simply leave a comment on blog and/or tweet about which variation you think increased conversions by 400%. If you tweet, either use the tweet boxes belows or make sure you cc @wingify so that we can track your entry.
When will be the winner chosen?: on September 8th, along with the actual case study, we will announce the winner of this contest.
About this A/B test case study: one variation had a Security badge in the sidebar and in the other there was no such badge. One of the variations saw an increase of 400% in conversion rate (click on coupon code).
So, which version increased conversions by 400%?
– Version A: with Security badge in sidebar –
– Version B: without Secure badge in sidebar –
Remember, the prize for this contest is a full year of $249/mo Visual Website Optimizer subscription total worth $3000! So, enter take part into contest now either by tweeting your choice above or leaving a comment below. (Bonus: if you do both, your entry will be counted twice.)
