The Next Generation of Visual Website Optimizer is launching in April 2014 See What's Coming

A/B test case study: how two magical words increased conversion rate by 28%

Posted in A/B Split Testing on May 26th, 2010

When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.

Background

Soocial is an online address book that helps you keep your phone, computer and online services contacts sane. The app syncs, merges and provides backups so that you are never stranded without the contact details you need. It works with over 500 phone models, webmail (like Gmail, Hotmail and Yahoo) and your Mac and Outlook. Soocial chose to use Visual Website Optimizer for their testing needs. Their first test was to increase the click-through rate on the homepage to the signup form.

What was tested

Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.

Results

As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.

Control: 14.5% conversion rate

Variation: 18.6% conversion rate

The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.

Why “It’s Free” Worked

When we asked Soocial why they thought the winning variation worked, this is what they had to say:

Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.

So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:

To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.

For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.

How valuable was Visual Website Optimizer for the A/B Test?

Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:

Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.

Key Takeaways

Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!

Paras Chopra

CEO and Founder of Wingify by the day, startups, marketing and analytics enthusiast by the afternoon, and a nihilist philosopher/writer by the evening!

The Complete A/B Testing Guide

Know all that you need to get started:

  • What is A/B Testing?
  • Is it compatible with SEO?
  • How to start your first A/B test?
Show Me The Guide


Tags

42 Comments
Matt S
May 26, 2010

An interesting case study.

I have a little qualm with this particular in that essentially it has been used to show that misleading the customer, however slightly, was more effective than not doing so.

It may not be a huge deal but to me part of CRO is helping and assisting the customer towards the conversion goal rather than tricking them towards it. In this case it’s barely an issue for most but I bet some would be put out when they discover that they have to pay because they have 250+ contacts.

Just a thought.

Greg
May 26, 2010

Well, right in your first paragraph when you said “can you guess which one did the trick?”, I knew it was “It’s free”.

It’s easy to understand : “Signup for free” or “free signup” implies something like “it’s free to get in, but then you’ll have to pay for the product to be of any use”. Where as “It’s free” means “It’s free, you won’t have to pay anything”

Matthew
May 26, 2010

What was the sample size?

Was the difference in conversion rates statistically significant? at what confidence level?

:)

Paras Chopra
May 26, 2010

Matthew, the difference in conversion rate was significant at 95% confidence level. A lot of our customers choose not to reveal sample size and traffic.

[...] full post on Hacker News If you enjoyed this article, please consider sharing it! [...]

Chris M
May 26, 2010

What an excellent case study, it goes to show how it’s the small things that count! What I’d like to know is whether the main aim was to just get people to sign up, because then the goal was achieved, but in terms of long term results, a lot of the created accounts will probably be inactive, a lot of people just sign up for the sake of it when “free” is mentioned..

Dee kumar
May 26, 2010

The key lesson here, which I often repeat on my blog is split test and observe the difference.

It’s interesting how a simple change of words made all the difference with people signing up, but it would also be interesting to observe which set of customers purchases more also. Are the ‘it’s free’ customers expecting a free ride, where as the ‘sign up free’ people are expecting an upsell…

If you were my client, that is what I would want you to test next. There are also many other variables that could come into play such as day of the week and where the advertising took place.

But, kudos on a good mini case study on this split test.

Niels Bom
May 26, 2010

Aren’t those three words? (not a native speaker btw)

[...] this article: A/B test case study: how two magical words increased conversion rate by 28% « I love split testing … 26 May 2010 | Uncategorized | Trackback | del.icio.us | Stumble it! | View Count : 0 Next Post [...]

Keywords Consultants
May 26, 2010

Funny how the simple things bring the best results. Interesting article, keep them coming :)

[...] Twitter’s New Ad Policy And The Dangers Of Tying Your Business To Another A/B test case study: how two magical words increased conversion rate by 28% [...]

[...] Twitter’s New Ad Policy And The Dangers Of Tying Your Business To Another A/B test case study: how two magical words increased conversion rate by 28% [...]

HowToPlaza
May 27, 2010

I think what would give more positive result is if you specify how many records are actually free. Free often still means you can sign up for free but you need to pay to use services, and many users are aware of this trap.

Sandra
June 2, 2010

Of course one can increase the number of sign-ups by lying – but do you really want to do this?
You would act like a person with dubious moral standards and you have potential legal problems at hand.
The art is to increase your sign-ups without being dishonest, not increasing the number of sign-ups by promising stuff you will not deliver.

[...] Two Magical Words Increased Conversion Rate by 28% The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results. [...]

[...] Two Magical Words Increased Conversion Rate by 28% The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results. [...]

[...] Two Magical Words Increased Conversion Rate by 28% The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results. [...]

[...] Two Magical Words Increased Conversion Rate by 28% The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results. [...]

[...] Two Magical Words Increased Conversion Rate by 28% The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results. [...]

[...] A/B test case study: how two magical words increased conversion rate by 28% [...]

[...] A/B test case study: how two magical words increased conversion rate by 28% [...]

Service Desk
July 23, 2010

The FREE word always comes in handy. It’s really what attracts people to sign in and join what we offer.

Optimizely
September 2, 2010

Excellent point Greg! It does not get much more simple and easy to understand than “It’s free”. It’s interesting to see how far simplicity can go in increasing conversion rates!

AJ
September 9, 2010

What’s the p-value of your test?

[...] Soocial added the words “it’s free” next to the main CTA button and saw conversions rise 28%. (Source) [...]

[...] to help you decide which to use. Sounds pretty simple, right? Well, in some ways, it is. Often, even a simple change can make a drastic difference in how your customers perceive your website, and can lead to dramatic [...]

[...] Soocial: how two magical words increased conversion rate by 28% [...]

[...] Two magical words improve conversion rate by 28% [...]

[...] charge it is often beneficial to remind them that they’re getting something for free. A Visual Website Optimizer case study saw a 28% conversion rate lift for online address book Soocial simply by adding the words “It’s [...]

[...] what if you want to test adding just a couple of words to your call to action as Soocial did in this example. Adding two words beside the CTA button increased conversions on this page by 28%. Luckily the [...]

[...] Even adding two simple words near your call to action could have dramatic impacts on your conversion rate, as was found out in this A/B test. Read the full case study here. [...]

[...] Two Magical Words Increased Conversion Rate by 28% [...]

[...] 20% ImprovementSimple Call to Action Increases Twitter Clickthroughs – 12.81% ImprovementChanging Two Words Increased Conversion Rates – 28% ImprovementDailyBurn Google Website Optimizer Case Study – 29.6% [...]

Selena Gomez
January 3, 2012

It’s fairly obvious why adding the phrase “It’s free” helped improve conversion rates – the visitor is excited that the service/product/etc is free. However, they may become frustrated if they learn they have to pay somewhere along the line.

balaji
January 4, 2012

Its works…we change our call to action button same as here…then the conversion increase 25%

UX Designer
January 5, 2012

[...] Two Magical Words Increased Conversion Rate by 28% The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results. [...]

[...] Soocial added the words “it’s free” next to the main CTA button and saw conversions rise 28%. (Source) [...]

[...] Due parole magiche che consentono di migliorare il tasso di conversione del 28% [...]

[...] two magical words added next to the “Sign Up Now” button on Soocial’s website improved conversion [...]

[...] 2 marvelous phrases additional close to the particular “ Sign Up Now” button upon Soocial’ nasiums [...]

Leave a comment
Required
Required - not published


− 6 = three

Notify me of followup comments via e-mail.

RSS feed for comments on this post. TrackBack URL

I ♥ Split Testing Blog


Stay up to date

Recent Posts

Write for this blog!

We accept high quality articles related to A/B testing and online marketing. Send your proposals to contribute@wingify.com