People usually have lots of unfounded excuses for not doing A/B testing. A major excuse right on top of the list is ”OMG, WE DONT WANT ANY LOST SALES”. Its toned down interpretation means if test variations are not good enough, A/B testing can actually hurt a website’s conversion rate. This reason is a major deterrent for eCommerce (or other kinds of) websites where every sale matters. So, if a variation isn’t performing as well as the control (or deafult), that means sales are getting lost during A/B testing. On the face of it, the argument sounds really convincing.
However, the reasoning is quite weak, and that is because:
a) Your benchmark should be overall site conversion rate, not the conversion rate of control
Poorly performing variations is the price you pay for doing A/B testing. That said, if some variations aren’t doing well, other variations may be doing much better. Even if multiple variations perform badly but your overall conversion rate is good (perhaps due to one or two extremely well performing variations), you should be in your comfort zone. Actually, the fear of poorly performing variations resulting in lost sales is completely irrelevant as long as overall conversion rate (accounting all variations) is higher than your existing (control’s) conversion rate. So, before starting to worry take a peek at the right metrics
b) A secret ninja trick: you can always disable poorly performing variations
Most modern A/B testing tools (including Visual Website Optimizer) provide one-click option to disable variations that you don’t want to keep in the test. If you think a particular variation is not performing well, simply disable it. (Remember: don’t disable a variation at the slightest hint of poor performance – always wait for statistical significance).
c) Still not convinced? You can always let the Visual Website Optimizer monitor the test for you
Here comes the magic part. Visual Website Optimizer has recently introduced a feature called “Risk-Free Testing” whereby (if you choose) your test is monitored in the background and VWO automatically disables poorly performing variations without any human intervention. That means your test is guaranteed to perform as well as your website’s existing conversion rate because all non-performing variations get disabled as soon as they are detected .
d) Okay, you even want maximum conversion rate possible? Sir, you can implement the winning variation on your site automatically
Another part of “Risk-Free Testing” is an option to automatically allocate 100% traffic to the winning variation once a winning variation has been found (during monitoring in background). This dramatically reduces the amount of attention you need to devote to the test as it ensures that the winning variation will be implemented on your website automatically. No IT department involved for post-test implementation and your developers can go on a much needed vacation.
e) In nutshell, your only risk is the time invested. But, hey, look at the upside.
As argued above, “Risk-Free Testing” removes all risk involved in doing an A/B test. Given zero risk, what is the worst that can happen if you do an A/B test? Your time invested can get wasted? Yes, your 20 minutes invested in thinking a new headline or 2 days redoing the sales page can indeed get wasted. However, look at the potential upside: 20% increase in sales, 90% increase in conversions and what not.
By the way, a “no-result” isn’t worthless – it tells that your existing page is doing really doing well and you should avoid tinkering with it. Or it says you are testing wrong sections or your variations are poor. It is actually a lot of information and should give you interesting insights for your next A/B test.
To sum this rather long post, here is a golden nugget for you: What cannot come down can only go up. (And, yes, we are talking about conversion rate here)
A picture is worth a thousand words. And a video is worth laboring through hundreds of documentation pages and forum threads. The following video shows how to create a multivariate test using Visual Website Optimizer in well under 4 minutes. The short tutorial is a must watch if you have heard about multivariate testing in articles/blogs; and always wondered what exactly is it. Even if you haven’t heard about multivariate testing, don’t worry – this tutorial with demystify that. Basically, multivariate testing is a technique where you vary several elements on your page to see which version results in maximum conversions (that is, form submissions in this case). A successful multivariate test can often result in 50-100% increase in conversions (case study to come out soon).
Without any further blabbering, here is your video tutorial on multivariate testing (click Play to start the video):
This post is a compilation of best A/B split testing case studies available on the Internet. In a way, this compilation proves that: a) A/B split testing really works; b) it doesn’t take much effort (hello Visual Website Optimizer); c) you should be doing it right now! I think case studies are an interesting way to learn about a new domain. So, consider this post as a starting point for the wonderful world of A/B split testing. Let’s start the list of case studies which you should definitely read today.
Writing Decisions: Headline tests on the Highrise signup page
37Signals did an A/B test on the headline of their pricing page. What they found was that “30 day Free Trial on All Accounts” had 30% more signups than the original “Start a Highrise Account”.:
Lesson: in a headline, the word “free” coupled with specific benefits (“30 day trial” v/s “start an account”) can do wonders.
Dustin Curtis’ “You should follow me on Twitter here”
The much hyped split testing case which involved testing of multiple different versions of call to text for twitter followers. Dustin found out that “You should follow me on Twitter here” worked 173% better than his control text “I’m on Twitter”. Though his results were fantastic, I recently wrote a post titled “You should NOT follow me on Twitter” arguing why you should be wary of applying this results without testing on your website.
Lesson: persuasive call to action “you should follow” proves to be much better than a passive call to action “i’m on twitter” (well, why would a visitor care).
A simpler variation which had less options for a visitor to choose from resulted in 20% increase in conversions. The winning version was also much easier on eyes (in terms of details and text) as compared to the control.
Lesson: too many options on a page usually overwhelms the visitor and s/he simply bounces off the page.
A/B Test Case Study: Single Page vs. Multi-Step Checkout
Reducing a two page checkout process to one page checkout increased conversions by 20%. Though overall increase in sales and orders didn’t increase much, checkout % did increase.
Lesson: lesser the steps you have in your funnel, higher will be your conversion rate. A lot of visitors drop off at each stage of conversion funnel so try to shorten it as much as possible (ideally to one page).
How To Increase Your Adsense Earnings 94% Overnight
Sensationalist headline but interesting results. hey tried different combinations – ads in left v/s right; ads top v/s bottom; test ads v/s image ads. Not sure if the results are statistically significant, but definitely a good case study.
Lesson: proves that A/B testing not just works for product/sales websites, but can work for advertisement supported pages too.
Marketing Experiments Response Capture case study – triple digit increase in conversions
They demonstrated a 258% in opt-in rate. Focus was to remove all distractions and require the visitor to only provide email address. For completing his/her complete profile, the landing page incited the visitors with an Amazon gift card (which was again split tested).
Lesson: require the visitor to input minimal amount of information on the landing page, you can always ask for more information later (it related is the concept of sunk cost).
How we increased the conversion rate of Voices.com by over 400%
In addition to A/B split testing, they used a lot of other tools/techniques – clickmaps/user feedback/usability testing. The final result was an increase in 400% increase in conversions.
Lesson: simplicity; social-proof; role of video in conversions; clarity in design
Skype Boosts Homepage Conversion Rate with Radical Simplicity
Skype tested their homepage for 3 different variations. Key difference between the three versions was the level of details presented on the page. Guess which variation won the test? The simplest variation of all (see image above) saw an increase of 5% in downloads. The figure may not look like much of an improvement but when you consider that Skype may be getting tens of thousands of downloads each day, this increase becomes quite large and totally worth-it spending thousands of dollars on doing the A/B test. (But, hey, you don’t have too – after all, Visual Website Optimizer is free to signup).
Lesson: simplicity, simplicity and some more simplicity
Do you know of any other interesting A/B split testing case studies which I should add in the post? Perhaps you may want to take help of A/B ideafox (a search engine for A/B split testing case studies).
