It is a myth that A/B testing can only be used for optimization and you need to fully establish a process optimizing it. In other words, many people think that A/B testing is best meant for big guys who already have an established conversion funnel, and now want to squeeze an extra 0.01% out of it. As a natural consequence to this logic, it may appear that A/B testing cannot be used at initial stages of a project (or startup) because there is nothing to optimize yet.
Wrong. There are actually two distinct uses of A/B testing: a) optimization; b) validation.
How to use A/B testing for Optimization?
As mentioned in the opening paragraph, this is the activity with which A/B testing is usually associated. Typical scenario: you are running an advertising campaign where you collect leads. You get 10 leads for every 100 visits and you realize at this conversion rate (10%), this activity isn’t profitable. So, you set out to optimize (or increase) the conversion rate. You test headline, product image, form length and a bazzilion different page elements. In the end, you manage to increase the conversion rate to 15%. Well done!
As you can see, with this A/B test, the objective was clear: optimize conversion rate for the activity one is currently doing. There are several Visual Website Optimizer case studies where the tool was used for optimization. Examples include increasing sales by 20% by testing page design and increasing sales by 6.3% by testing Buy Now button.
How to use A/B testing for Validation?
Unlike above where we answer what is the best way to do an activity, in this case A/B testing is used to answer which activity to do in the first place. Validation means coming up with different possibilities and testing which one works best. Typical scenario: you decide to publish a new whitepaper for gathering leads. However, you are unsure which whitepaper will be most interesting to your customers. To answer that question, you make 5 different landing pages for each whitepaper and setup an A/B test. After the test is over, you check which whitepaper garnered most interest (via pre-release signups or some other conversion metric). Notice that you aren’t optimizing the lead collection process here, rather you are testing which whitepaper will potentially collect most leads.
To give you some examples, there are several Visual Website Optimizer case studies where validation was carried out. Here is how a startup validated its positioning and here is how the appropriate position for a promotional message was tested.
Key Lesson
A/B testing isn’t just for big corporations who want to optimize the last drop out of their conversions. In fact, A/B testing is more useful at answering business questions. Want to know which product to launch next? Which feature to develop? Which new market to enter? A/B test it today!
What is your opinion about this? Did you ever use A/B test for validation, instead of optimization?
RIPT Apparel is a Chicago-based online retailer of one-of-a-kind designer tees and wearable art. They sell one new design by a new artist every day starting at midnight CST for $10 and it is only available for a period of 24 hours. After that period, the design rests in peace forever in the T-shirt graveyard and a new one takes over its place. The sense of urgency created by limited availability of the design can be used effectively to drive sales and that is what RIPT Apparel did in their first A/B test.
They had never done A/B testing before but were intrigued by the concept and wanted to give it a shot. The tool they selected for this purpose was Visual Website Optimizer. They tested their current buy button against a new buy button to determine which would have the best conversion rate. The conversion rate here being sales of their T-shirts.
The following is their control version (or default). Observe that the button isn’t the first thing to catch your attention as it blends well into the overall color scheme.
Original Version
They saw an opportunity for improvement, so they replaced their existing button with an attention-grabbing button with a different color scheme (green). See version A of the button below:
Version A – Control
Much to their surprise, they immediately saw a rise in sales. Ideally, they should have A/B tested the new version against the old but this was their first time and as you will read further they did a great job overall. Encouraged by the results, they went ahead and created another variation of the button. This time they decided to emphasize on the tactic that they should have used all along: emphasizing “limited 24-hour availability”. It has been demonstrated again and again that a sense of urgency drives up the conversion rate and this time too it proved to work best. Following version (B) of the button increased their site sales by 6.3% (notice only the button was changed – no new offers, no new products, no new policies).
Version B – 6.3% increase in sales
Pause for a moment and think what really happened in this case study. Someone (RIPT Apparel) with no previous experience in the field decides to A/B test, spends time researching what could work better, designs button variations, uses Visual Website Optimizer and sees a 6.3% rise in sales. No gimmicks, no new offers, no change in company strategy. Simply testing of sensible better variations of buy now button.
That 6.3% rise in sales will be hopefully permanent, so this one time effort would pay back RIPT Apparel many times over for time to come. Here is what they had to say after the test ended:
“According to Visual Website Optimizer we found our conversion rate improved a whopping +6.3%! We were astonished by this result. We had no idea that changing one button could result in an increase in sales and have that much of an impact. We wish we found this tool much earlier. “
It is important not to forget the role of the tool here. Visual Website Optimizer has been designed with a sole focus on ease of use, while avoiding technical hassles as much as possible. When we asked about the role of the tool in their first A/B test, RIPT Apparel said:
“The sales we have seen since changing and experimenting with our buy button have proven Visual Website Optimizer invaluable. I had some issues setting it up due to our shopping cart system and the team at Visual Website Optimizer walked me through step by step in order to set it up correctly.
As a result of this test I was able to go back to my partners and show them the results, which they were very happy with. Version B. is now live on our site and sales continue to rise.”
A great thing about tasting success with A/B testing is that it pumps you up to do even more A/B testing. So, RIPT Apparel are all set for their next phase of testing:
“We learned to test and test often. We think now that this test is over, we are going to try and test a few different color options, maybe even try some wording choices for our buy button.”
RIPT Apparel is a great company that successfully proved that A/B testing is not something that only Fortune 500 companies can do. Even small-medium sized online business can optimize and increase their sales by not mistaking A/B testing as a hyped-up fad. A/B testing works and it is addictive; one simply needs to try it at least once.
By the way, you should check out RIPT Apparel’s T-shirts – they are really quite cool!
This post is a compilation of best A/B split testing case studies available on the Internet. In a way, this compilation proves that: a) A/B split testing really works; b) it doesn’t take much effort (hello Visual Website Optimizer); c) you should be doing it right now! I think case studies are an interesting way to learn about a new domain. So, consider this post as a starting point for the wonderful world of A/B split testing. Let’s start the list of case studies which you should definitely read today.
Writing Decisions: Headline tests on the Highrise signup page
37Signals did an A/B test on the headline of their pricing page. What they found was that “30 day Free Trial on All Accounts” had 30% more signups than the original “Start a Highrise Account”.:
Lesson: in a headline, the word “free” coupled with specific benefits (“30 day trial” v/s “start an account”) can do wonders.
Dustin Curtis’ “You should follow me on Twitter here”
The much hyped split testing case which involved testing of multiple different versions of call to text for twitter followers. Dustin found out that “You should follow me on Twitter here” worked 173% better than his control text “I’m on Twitter”. Though his results were fantastic, I recently wrote a post titled “You should NOT follow me on Twitter” arguing why you should be wary of applying this results without testing on your website.
Lesson: persuasive call to action “you should follow” proves to be much better than a passive call to action “i’m on twitter” (well, why would a visitor care).
A simpler variation which had less options for a visitor to choose from resulted in 20% increase in conversions. The winning version was also much easier on eyes (in terms of details and text) as compared to the control.
Lesson: too many options on a page usually overwhelms the visitor and s/he simply bounces off the page.
A/B Test Case Study: Single Page vs. Multi-Step Checkout
Reducing a two page checkout process to one page checkout increased conversions by 20%. Though overall increase in sales and orders didn’t increase much, checkout % did increase.
Lesson: lesser the steps you have in your funnel, higher will be your conversion rate. A lot of visitors drop off at each stage of conversion funnel so try to shorten it as much as possible (ideally to one page).
How To Increase Your Adsense Earnings 94% Overnight
Sensationalist headline but interesting results. hey tried different combinations – ads in left v/s right; ads top v/s bottom; test ads v/s image ads. Not sure if the results are statistically significant, but definitely a good case study.
Lesson: proves that A/B testing not just works for product/sales websites, but can work for advertisement supported pages too.
Marketing Experiments Response Capture case study – triple digit increase in conversions
They demonstrated a 258% in opt-in rate. Focus was to remove all distractions and require the visitor to only provide email address. For completing his/her complete profile, the landing page incited the visitors with an Amazon gift card (which was again split tested).
Lesson: require the visitor to input minimal amount of information on the landing page, you can always ask for more information later (it related is the concept of sunk cost).
How we increased the conversion rate of Voices.com by over 400%
In addition to A/B split testing, they used a lot of other tools/techniques – clickmaps/user feedback/usability testing. The final result was an increase in 400% increase in conversions.
Lesson: simplicity; social-proof; role of video in conversions; clarity in design
Skype Boosts Homepage Conversion Rate with Radical Simplicity
Skype tested their homepage for 3 different variations. Key difference between the three versions was the level of details presented on the page. Guess which variation won the test? The simplest variation of all (see image above) saw an increase of 5% in downloads. The figure may not look like much of an improvement but when you consider that Skype may be getting tens of thousands of downloads each day, this increase becomes quite large and totally worth-it spending thousands of dollars on doing the A/B test. (But, hey, you don’t have too – after all, Visual Website Optimizer is free to signup).
Lesson: simplicity, simplicity and some more simplicity
Do you know of any other interesting A/B split testing case studies which I should add in the post? Perhaps you may want to take help of A/B ideafox (a search engine for A/B split testing case studies).
