In a previous post, I provided a downloadable A/B testing significance calculator (in excel). In this post, I will provide a calculator which lets you estimate how many days should you run a test in order to obtain statistically significant results. But, first, a disclaimer.
When someone asks how long should s/he run an A/B test, the ideal answer would be until eternity or till the time you get results (whichever is sooner). In an A/B test, you can never say with full confidence that you will get statistically significant results after running the test X number of days. Instead, what you can say is that there is 80% (or 95%, whatever you choose) probability of getting statistically significant result (if it indeed exists) after X number of days. But, of course, it may be the case that there is in fact no difference in performance of control and variation so no matter how long you wait, you will never get a statistically significant result.
Download and use the calculator below to find out how many visitors you need to include in the test. There are 4 pieces of information that you need to enter:
Once you enter these 4 parameters, the calculator below will find out how many visitors you need to test (for 80% and 95% probability of finding the result). You can stop the test after you test those many visitors, but you should never stop earlier than that. You may end up concluding wrong results.
Click below to download the calculator:
Download A/B testing duration calculator.
Please feel free to share the file with your friends and colleagues or post it on your blog / twitter.
PS: By the way, if you want to do quick calculations, we have a version of this calculator hosted on Google Docs (please make a copy of the Google Doc sheet into your own account before you make any changes to it).
Ah! The million dollar calculator. Explaining how it works is beyond the scope of this post as it is too technical (maybe a separate post). But, if you have got stomach for it, below is gist of how we calculate number of visitors needed to get significant results.
The graph above is taken from an excellent book called Statistical Rules of Thumb. Luckily, the chapter on estimating sample size is available to download freely [PDF]. Another excellent source to get more information on sample size estimation for A/B testing is Microsoft’s paper: Controlled Experiments on the Web: Survey and Practical Guide [PDF].
Hope you like the calculator and related resources. Excited to know your feedback and comments!
Hyundai, the Korean giant, not just makes well-engineered cars, they have well-engineered online strategy as well. In the Dutch market, they hired one our certified partners, Traffic4U to optimize conversion rate on their lead-generation pages. Traffic4u is an experienced, international online marketing agency, specializing in Result Driven Online Marketing and since they are our partners, they chose Visual Website optimizer for testing and optimization on Hyundai.nl
In this case study, we catch up with Janco Klijnstra from Traffic4U who talks about how they used Visual Website Optimizer for increasing conversion rate by 62% for Hyundai.
Hyundai has landing pages for all of their car models where people can request for a test drive or download a brochure. These landing pages mainly get traffic from paid advertising campaigns (but get some direct and SEO traffic too). They wanted to make most from this traffic hence partnered with Traffic4U for conducting a multivariate test on all car model pages.
There were several goals for this test that they wanted to optimize:
The main difference between A/B testing and multivariate testing is that in A/B testing variety of changes are done in a single variation while in multivariate testing, every change you make creates a new variation to be tested. So, multivariate testing is useful if you have multiple ideas to test on a page as it tells you exactly which changes impact your conversion rates and which don’t.
Hyundai’s car landing pages have a lot of different elements (car headline, car visuals, description, testimonials, etc.) so it was essential to setup a multivariate test in order to understand which elements influence a visitor’s decision to ask for a test drive or download a brochure.
Traffic4U setup a multivariate test and decided to create variations of following sections of the page:
A total of 8 combinations (3 sections, 2 variations each = 2*2*2) were generated for this multivariate test. Here’s screenshot of the original page:
- Original page -
The results of this multivariate test were phenomenal. One of those variations increased conversion rate (request for test drive or brochure) by 62%. And there was a staggering 208% increase in clickthrough rate (step 1 to step 2). Out of the total 8 combinations, can you guess which one increased conversions?
Well, the combination with SEO text, extra buttons and larger images did the wonder. It’s amazing how Traffic4U was spot on for all three changes. Here’s how the variation looked like:
- Winning variation (62% increase in leads) -
The results of this multivariate test are so phenomenal that it won a silver award in annual WhichTestWon awards!
Traffic4U used lot of heuristics and best practices experience for coming up with variations for this test (larger pictures, clear call to action buttons, etc.). This test demonstrates that doing a proper multivariate test can validate those best practices. It may have been certainly possible that adding SEO text decreased conversions at the expense of extra traffic, however setting up a multivariate or A/B test can settle that dilemma easily.
When we asked Traffic4U about their comments on Visual Website Optimizer, here’s what they had to say:
Visual Website Optimizer was very valuable it made it possible for us as an agency to build our own variations, test on a group of pages and measures different goals at once. Also the Analytics plug-in was very helpful in the analysis for segmentation purposes.
This case study is an excellent example of how a large company (such as Hyundai) works with a specialized conversion rate optimization agency like Traffic4U to use multivariate testing for increasing business metrics such as test drive requests and brochure requests. If it worked for Hyundai, it can work for you too!
Signup for a free 30 day trial account of Visual Website Optimizer to easily setup multivariate tests on your website to increase sales and conversions.
In the past, we have featured a number of A/B testing case studies. In many of those case studies, a number of elements change which contribute to increased sales and conversions. However, this case study is special. Here a single (seemingly trivial) change was responsible for increasing sales by as much as 34%. This is unprecedented and, of course, is an amazing story which demonstrates the power of A/B testing.
That’s why I chatted with Chris Muktar of WikiJob.co.uk, UK’s largest graduate jobs website. Started in 2007, it now attracts over 500,000 visits a month. Started by two friends, it has grown to be a profitable company employing several staff, and proudly boasts many of the world’s largest brands as their clients.
WikiJob use Visual Website Optimizer for A/B testing and below is the interview where Chris talks about how they managed to boost bottomline sales by 34% just by testing a small element.
What conversion goals were measured in the test?
There were two conversion goals. The first was to click through to Paypal checkout. The other was to complete it and actually buy something.
On which page did you run the test?
http://www.wikijob.co.uk/aptitude-tests
What is the traffic source? Organic, direct, PPC, etc.?
Most of the traffic was from our main http://www.WikiJob.co.uk website. The traffic to our website is almost entirely organic.
Which part of page did you select for the test and what variations did you test?
We’ve tested a number of things in the past and had some great results. This time we were testing the effect of three lines of testimonials given by customers and their impact on sales. Below are screenshots.
- Original Page - 
- Variation (with screenshots): 34% increase in sales - 
Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
A previous test displayed the same testimonials further down the page and had no discernable impact on conversion. However, it’s commonly thought that social proof can help in the sales process, so we wanted to leave nothing to chance and test it.
What results did you get? Were you surprised by the results?
Testimonials increased sales by 34%. The testimonials we used are very ‘sober’ (compared with the overly enthusiastic ones you so often see in marketing literature). The test results were surprising. Although such increases of sales can be quite normal in split testing, I did not think that testimonials would make such a difference (and indeed put off testing them, thinking they were irrelevant). The increase in revenue was very substantial.
Any lessons which can be derived from your test?
Social proof is important, and it is important to test everything. I may try putting some small pictures next to each testimonial to see if this improves things. [Editor's note: do that! Our previous split tests show that human pictures can potentially double conversion rates]
How valuable was Visual Website Optimizer for this test?
Visual Website Optimizer has been invaluable. You pay for the service once, but the increase profits and revenues continue into perpetuity. Through the use of several iterations of tests, we have increased sales threefold.
Visual Website Optimizer is expanding its team! I am so proud of the progress we made in last 10 months (we launched paid plans in May 2010): growing from 0 to 7000+ users we have today. We are entirely self-funded (have not raised any angel or VC round) and have been profitable from day 1. Thank you, our lovely customers.
Touch wood, things have gone well for us and I hope they will continue to be good. That’s why we have now decided to move to a new office and grow the team by hiring the best programmers, support engineers and sales guys we can find in Delhi, India. (Already extended offers to 2 more kickass engineers.)
Our new office is on 7th floor of a multi-storied building and has a view that is unmatched with any other office place we have seen. We simply fell in love with it when we were looking for a new place. See some of our pics below:
That’s me (Paras Chopra) and my new workspace!
Our team (from left to right): Sushaantu, Sparsh and me (Paras)
Another beautiful view (Sushaantu in the foreground, Sparsh is the cameraman!)
We recently put up a careers page and are looking forward to hiring people for three profiles below:
We prefer candidates who are willing to work from our office in Delhi, India (we prefer having team at one physical location though for customer happiness and technical support, we are willing to work with remote candidates). So if you are interested in joining a bootstrapped, fast growing and profitable startup or know someone who could be interested, please do let us know at careers@wingify.com.
We’re hungry to grow more and more!
We already have powerful behavioral targeting and segmentation options in Visual Website Optimizer. Today, we’re proud to announce a brand new feature that will take your website to the next level. We just released a feature that is Holy Grail of targeting and personalization: knowing what a visitor is thinking while browsing your website (and then displaying relevant content to him/her to boost conversion rate and sales).
Yes, yes, yes, we are as excited about this new feature as you are. Once you install Visual Website Optimizer on your website, the mind reading feature uses subtle differences in how a visitor is moving his/her mouse to gauge what s/he is thinking right now. Essentially, you get a report on following four attributes (note that this is an initial list, it will expand as our research progresses):
We’re an A/B testing company and to prove this new feature works, we conducted an A/B test. Our hypothesis was that by changing a webpage we can affect visitor mood and if Visual Website Optimizer’s new feature detects that change, it definitely works. We did this test on our homepage. Here’s original homepage:
- Original Homepage – 
In variation, we tried replacing text with an image that was definitely going to affect visitor’s mood in some way. Here’s the variation:
- Variation of Homepage – 
We did this test for a little over a week and here are the results:
As expected, happiness and fun factor increased tremendously when visitors saw that poor boy’s image on our homepage. Moreover, people started subconsciously associating rain drops in the picture with likelihood that it is going to rain today.
Most surprisingly, there was a sharp drop in trust and total signup rate because of that poor boy on our homepage (Glad we have stopped the test. It affected us financially!) Why did that drop happen? We cannot tell! There are always somethings that remain a mystery after an A/B test is completed.
In a word: mathematics! We use various inputs to determine what a visitor is thinking. The most significant input is visitor mouse movements. The way people move their mouse tells a lot about what they are thinking. Strange but true!
Details of implementation only for nerds and geeks: we used artificial neural networks to train a predictive model whose input is vector comprising of mouse movements in past 30 seconds and whose outputs are various factors like happiness, raininess, etc. The data for training this model came from 6 month long research where we recruited a team of 10 normal people from 4Chan.org and asked them to visit various websites and tell us what they were thinking while they were browsing. Without their knowledge we recorded mouse movements and then simply used our expertise in machine learning to map mouse movements to behavior! Genius, isn’t it?
According to our knowledge, no other company in the world provides such amazing technology. So, why don’t you signup for a free 30 day trial of Visual Website Optimizer and try the new feature without any obligations? We can assure you that you are going to have a lot of fun spooking your visitors 
