Visual Website Optimizer

When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.

Background

What was tested

Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.

Results

As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.

Control: 14.5% conversion rate

Variation: 18.6% conversion rate

The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.

Why “It’s Free” Worked

When we asked Soocial why they thought the winning variation worked, this is what they had to say:

Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.

So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:

To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.

For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.

How valuable was Visual Website Optimizer for the A/B Test?

Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:

Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.

Key Takeaways

Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!

42 Comments »

Answer: none. Using Visual Website Optimizer on my blog, I tested the layout of sidebar to see if it has any effect on bounce rate. Particularly, I tested whether the sidebar in right (default on many blogs) works better than the sidebar on left. The answer for my case turned out to be that it doesn’t really matter. (Though it may differ for you – you should A/B test it, it is really quite simple). Here are the versions I tested:

Control – Sidebar on the right (click to expand)

Variation – Sidebar on the left (click to expand)

Both versions had surprisingly similar engagement rate of around 22%. Engagement rate is inverse of bounce rate, so in a way this means , irrespective of sidebar positioning, bounce rate of my blog remains 78% (quite high, by the way). If you want to see actual results, here they are (click to expand):

Since the test involved changing the layout of blog across all posts, if you are curious that this would have involved digging into WordPress or PHP code – the answer is no. In fact, the took just 5 minutes to implement. Using the advanced code mode of VWO, all I did was to define the Left Sidebar variation with this CSS: #sidebar { float: left; }. Yes, just that! VWO took care of the rest.

11 Comments »

RIPT Apparel is a Chicago-based online retailer of one-of-a-kind designer tees and wearable art. They sell one new design by a new artist every day starting at midnight CST for $10 and it is only available for a period of 24 hours. After that period, the design rests in peace forever in the T-shirt graveyard and a new one takes over its place. The sense of urgency created by limited availability of the design can be used effectively to drive sales and that is what RIPT Apparel did in their first A/B test.

The following is their control version (or default). Observe that the button isn’t the first thing to catch your attention as it blends well into the overall color scheme.

Original Version

They saw an opportunity for improvement, so they replaced their existing button with an attention-grabbing button with a different color scheme (green). See version A of the button below:

Version A – Control

Much to their surprise, they immediately saw a rise in sales. Ideally, they should have A/B tested the new version against the old but this was their first time and as you will read further they did a great job overall. Encouraged by the results, they went ahead and created another variation of the button. This time they decided to emphasize on the tactic that they should have used all along: emphasizing “limited 24-hour availability”. It has been demonstrated again and again that a sense of urgency drives up the conversion rate and this time too it proved to work best. Following version (B) of the button increased their site sales by 6.3% (notice only the button was changed – no new offers, no new products, no new policies).

Version B – 6.3% increase in sales

Pause for a moment and think what really happened in this case study. Someone (RIPT Apparel) with no previous experience in the field decides to A/B test, spends time researching what could work better, designs button variations, uses Visual Website Optimizer and sees a 6.3% rise in sales. No gimmicks, no new offers, no change in company strategy. Simply testing of sensible better variations of buy now button.

That 6.3% rise in sales will be hopefully permanent, so this one time effort would pay back RIPT Apparel many times over for time to come. Here is what they had to say after the test ended:

“According to Visual Website Optimizer we found our conversion rate improved a whopping +6.3%! We were astonished by this result. We had no idea that changing one button could result in an increase in sales and have that much of an impact. We wish we found this tool much earlier. “

It is important not to forget the role of the tool here. Visual Website Optimizer has been designed with a sole focus on ease of use, while avoiding technical hassles as much as possible. When we asked about the role of the tool in their first A/B test, RIPT Apparel said:

“The sales we have seen since changing and experimenting with our buy button have proven Visual Website Optimizer invaluable. I had some issues setting it up due to our shopping cart system and the team at Visual Website Optimizer walked me through step by step in order to set it up correctly.

As a result of this test I was able to go back to my partners and show them the results, which they were very happy with. Version B. is now live on our site and sales continue to rise.”

A great thing about tasting success with A/B testing is that it pumps you up to do even more A/B testing. So, RIPT Apparel are all set for their next phase of testing:

“We learned to test and test often. We think now that this test is over, we are going to try and test a few different color options, maybe even try some wording choices for our buy button.”

RIPT Apparel is a great company that successfully proved that A/B testing is not something that only Fortune 500 companies can do. Even small-medium sized online business can optimize and increase their sales by not mistaking A/B testing as a hyped-up fad. A/B testing works and it is addictive; one simply needs to try it at least once.

By the way, you should check out RIPT Apparel’s T-shirts – they are really quite cool!

14 Comments »