Visual Website Optimizer

When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.

Background

What was tested

Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.

Results

As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.

Control: 14.5% conversion rate

Variation: 18.6% conversion rate

The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.

Why “It’s Free” Worked

When we asked Soocial why they thought the winning variation worked, this is what they had to say:

Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.

So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:

To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.

For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.

How valuable was Visual Website Optimizer for the A/B Test?

Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:

Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.

Key Takeaways

Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!

42 Comments »

These days you have all sorts of trust badges available to put on your websites. Some are SSL badges, some are good business practices badges and some are even carbon neutral certification badges. But do these badges have any effect?

A Visual Website Optimizer user, Karsten Lund (a conversion specialist from Optimeria.com), recently carried out a six-month long A/B test to determine whether an E-mark badge (certifies ethical conduct of Danish businesses) impacts conversion rate. He wrote a detailed case study in Danish and we recommend you to read it here (if you understand the language). For those who prefer to read the case study in English, read this post instead.

What was tested?

Karsten tested the presence of E-mark (the eye-like icon in the image above) on an eCommerce store House of Kids. 50% visitors saw this badge and 50% didn’t. The test started producing great results right from the start but our team consulted Karsten to run the test longer so that he can be really sure about the significance of the results.

What was the result?

Yes, you read it right, a 32% increase in conversions with the E-mark badge. And Karsten ran the test long enough to make sure result was statistically significant.

Interesting nuggets

• Karsten observed that if the security badge requires a user to click, it actually decreases conversions because s/he is taken away from the website and is not able to complete the purchase. So, the 32% increase in conversions if of non-clickable badge.
• Karsten observes in the case study that Visual Website Optimizer is much, much better and easier than Google Website Optimizer. And, no, we didn’t pay him for saying this A lot of other users agree with him, though.

Grain of Salt

Even though this particular result has been fantastic, you shouldn’t adopt trust badges on your website without testing them first. Different types of websites/badges combinations may have different results. Elsewhere on Internet, some report an increase in sales, some report a decrease in sales. So, the best strategy is to test it first and then only implement it on your website.

1 Comment »

Answer: none. Using Visual Website Optimizer on my blog, I tested the layout of sidebar to see if it has any effect on bounce rate. Particularly, I tested whether the sidebar in right (default on many blogs) works better than the sidebar on left. The answer for my case turned out to be that it doesn’t really matter. (Though it may differ for you – you should A/B test it, it is really quite simple). Here are the versions I tested:

Control – Sidebar on the right (click to expand)

Variation – Sidebar on the left (click to expand)

Both versions had surprisingly similar engagement rate of around 22%. Engagement rate is inverse of bounce rate, so in a way this means , irrespective of sidebar positioning, bounce rate of my blog remains 78% (quite high, by the way). If you want to see actual results, here they are (click to expand):

Since the test involved changing the layout of blog across all posts, if you are curious that this would have involved digging into WordPress or PHP code – the answer is no. In fact, the took just 5 minutes to implement. Using the advanced code mode of VWO, all I did was to define the Left Sidebar variation with this CSS: #sidebar { float: left; }. Yes, just that! VWO took care of the rest.

11 Comments »

RIPT Apparel is a Chicago-based online retailer of one-of-a-kind designer tees and wearable art. They sell one new design by a new artist every day starting at midnight CST for $10 and it is only available for a period of 24 hours. After that period, the design rests in peace forever in the T-shirt graveyard and a new one takes over its place. The sense of urgency created by limited availability of the design can be used effectively to drive sales and that is what RIPT Apparel did in their first A/B test.

The following is their control version (or default). Observe that the button isn’t the first thing to catch your attention as it blends well into the overall color scheme.

Original Version

They saw an opportunity for improvement, so they replaced their existing button with an attention-grabbing button with a different color scheme (green). See version A of the button below:

Version A – Control

Much to their surprise, they immediately saw a rise in sales. Ideally, they should have A/B tested the new version against the old but this was their first time and as you will read further they did a great job overall. Encouraged by the results, they went ahead and created another variation of the button. This time they decided to emphasize on the tactic that they should have used all along: emphasizing “limited 24-hour availability”. It has been demonstrated again and again that a sense of urgency drives up the conversion rate and this time too it proved to work best. Following version (B) of the button increased their site sales by 6.3% (notice only the button was changed – no new offers, no new products, no new policies).

Version B – 6.3% increase in sales

Pause for a moment and think what really happened in this case study. Someone (RIPT Apparel) with no previous experience in the field decides to A/B test, spends time researching what could work better, designs button variations, uses Visual Website Optimizer and sees a 6.3% rise in sales. No gimmicks, no new offers, no change in company strategy. Simply testing of sensible better variations of buy now button.

That 6.3% rise in sales will be hopefully permanent, so this one time effort would pay back RIPT Apparel many times over for time to come. Here is what they had to say after the test ended:

“According to Visual Website Optimizer we found our conversion rate improved a whopping +6.3%! We were astonished by this result. We had no idea that changing one button could result in an increase in sales and have that much of an impact. We wish we found this tool much earlier. “

It is important not to forget the role of the tool here. Visual Website Optimizer has been designed with a sole focus on ease of use, while avoiding technical hassles as much as possible. When we asked about the role of the tool in their first A/B test, RIPT Apparel said:

“The sales we have seen since changing and experimenting with our buy button have proven Visual Website Optimizer invaluable. I had some issues setting it up due to our shopping cart system and the team at Visual Website Optimizer walked me through step by step in order to set it up correctly.

As a result of this test I was able to go back to my partners and show them the results, which they were very happy with. Version B. is now live on our site and sales continue to rise.”

A great thing about tasting success with A/B testing is that it pumps you up to do even more A/B testing. So, RIPT Apparel are all set for their next phase of testing:

“We learned to test and test often. We think now that this test is over, we are going to try and test a few different color options, maybe even try some wording choices for our buy button.”

RIPT Apparel is a great company that successfully proved that A/B testing is not something that only Fortune 500 companies can do. Even small-medium sized online business can optimize and increase their sales by not mistaking A/B testing as a hyped-up fad. A/B testing works and it is addictive; one simply needs to try it at least once.

By the way, you should check out RIPT Apparel’s T-shirts – they are really quite cool!

14 Comments »

Most likely, A/B testing isn’t a must do activity for you.  After all, it requires spending some time thinking what to test and then some more time is required for coming up with interesting variations. (Luckily, VWO has removed all the technical and integration hassles so you save time there). All in all, it requires time and effort investment, whose value and returns may not be obvious to you.  Fully agree with you on that front.

However, for most people, the bottleneck is creating their first A/B test. The technique sounds hyped up, but it really isn’t and there is proof of even the first, really simple test yielding extremely positive results.  CityCliq, a service that helps local businesses have presence online, created their first test on VWO and saw (statistically significant) 90% increase in conversions. Point to note is that it was CityCliq’s first test, they tested a small headline change, saw results and they are now hooked to A/B testing.

Even when the value of A/B test is strikingly clear to me and companies like CityCliq, AquaSoft and MedaliaArt, there are many who somehow still manage to come up with excuses for not doing A/B tests. One by one, I tackle all excuses and try to convince why you should create your first A/B test right now! Following is the list of common excuses and my perfectly justified counter-excuses:

A/B testing takes too much time to setup (integration and technicalities)

False! It takes less than 10 minutes setting up the test. If you don’t buy this, watch this video or try setting up a test yourself (while keeping an eye on the clock).

A/B testing takes a lot of effort in coming up with different variations

While this is true (and entirely justified) for A/B tests where entire design is tested, this is certainly false wherein you are testing headlines, call-to-action and other small but significant elements on your website. With WYSIWYG editor and, making variations is as easy as working on Microsoft Word (now some may debate even this but I think Word is pretty easy to use).

There are no guarantees that I will get results with A/B testing

True, there are no guarantees. But then there are no guarantees for anything worth its salt. Clichés apart, unless you act on something you shouldn’t expect your sales and conversions to increase automatically.  A/B testing is no magic, to be sure. You cannot simply signup for VWO and expect sales to rise. However, if you aren’t doing A/B testing (or other doing business/product improvements) you can definitely expect sales to stay flat.

A/B testing will hurt my existing conversion rates

No it won’t. Most sophisticated A/B testing tools, including VWO, have options to switch off or disable variations that aren’t performing well. So, you don’t really have to worry that your existing conversion rates will take a huge hit in case variations are bad. You can setup a test, sit back, relax and go have a hot cup of coffe.

I know the truth (or, in other words, I already know what will work on my website)

No, you don’t. Trust me, you don’t. There are lots examples of A/B testing where the winning variation was an underdog. Except for systematic testing, you have no way for knowing what will work and what won’t. Of course, if you really know the truth, try doing A/B testing of your existing design against your hunch-based improved design and let your boss, clients, colleagues know how you always knew yours is better. (Warning: this will either shatter your status as an expert or take it to the next level!)

It requires a lot of traffic

Yes and no. The significance of your results depend on a lot of factors,  and not just only on amount of traffic. You can play around with this A/B testing traffic estimation calculator and actually see what amount of traffic do you really need for your test. In some cases even a traffic of 300-500 visitors may be good enough. (In other cases though, 10k visitors may not be enough). But if you are testing for dramatic changes on the site, you can expect to get statistically significant results with moderate amount of traffic.

I am not ready for it yet

Why? Have you ever thought when would you actually be ready? When you roll out the product? When you start getting a million visitors each month? When your customer base doubles? Frankly, A/B testing should be done at all the stages of evolution of a website – from startup to establishment. In fact, for startups A/B testing is even more important because in initial days there are so many questions to be answered related to business, market, product, etc.  A/B testing helps you find the answers you need. (Like it did for CityCliq).

The key lesson here is that if you have been procrastinating A/B testing for long, you should do it right now.  If you have any excuses for not doing A/B testing that I haven’t covered, leave a comment on this post. I will most definitely come up with a counter-excuse why you should be doing A/B tests all the time

No Comments »

Quick, imagine that you own a small local business, say an automotive repair shop, and you are looking forward to using Internet for getting more business. While researching for your options for tools or services, which of the following pitches is going to appeal you most?

• Businesses grow faster online!
• Online advertising that works!
• Get found faster!
• Create a webpage for your business

Having difficulty making your mind? You are not alone. CityCliq provides low-cost, search engine-optimized webpages for small and large businesses. As they provide a suite of tools to businesses for creating and managing their online presence, getting the positioning right is critical. They recently used Visual Website Optimizer to A/B test which product positioning works best. The conversion goal for this test was a click on their pricing plan. Implementation wise, this simple test was quick to setup (took them just 15 minutes to do so); major effort was invested in coming up with good candidates for positioning. They ran the test for 2 weeks on their homepage and were really happy with the results.

(Click to Expand)

Here is how different positioning statements fare against each other:

About the results, here is what CityCliq had to say:

“Create a webpage for your business was our last test tagline and eventually showed a 98% chance of beating the original. In retrospect, its success makes sense, as it’s perhaps the purest, most direct representation of our product. It didn’t surprise me, as user behavior seems to vary wildly on the web. Plus, there hasn’t been much research done on the behavior of our target audience, so these results were more informative than surprising. “

What they mean is that this A/B test yielded more than just an increase in conversion rate. The test informed them about the psychology of their target customer. This is a perfect example of how A/B split testing can be used to peek into the minds of customers to see misalignment of what they want and what they think you offer. Create a webpage for your business as the winning variation tells that CityCliq’s prospective customers are already aware of benefits of being online and in fact are looking for a tool make that process simpler. All other variations tried to pitch the advantages of being online, which clearly isn’t that appealing to the customers.

Another point to note is that all but the winning variation had an exclamation mark. It would be interesting if they follow up the test with two versions of the winning variation – one with exclamation mark and one without. After A/B testing was completed, CityCliq implemented the winning variation permanently on homepage (as you can see on screenshot) and subsequently saw a rise in user actions. Inspired by their first successful test, they hope to setup a lot of A/B split tests in near future.

12 Comments »

AquaSoft was founded in 1999 and is a pioneer in photo presentation software. Their products cover a wide range from slide show creation, desktop publishing, photo books, etc. They recently candidly shared details from their latest A/B testing project, where they used Visual Website Optimizer for increasing sales from their eCommerce site by 20%. Here are the details on the test as told by AquaSoft:

Question: What was the conversion goal of the test?
Answer:  The goal was to increase the sales on our product overview page. This page already had a higher conversion rate than the rest of our pages because people who visit this page already are interested in buying.

Question: On which page did you run the test?
Answer: Shop Overview (listing of all articles).

Question: Which part of page did you select for the test and what variations did you test?
Answer:

Original version (click to expand)

Redesigned version (click to expand)

Question: Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Answer:

• Clearer design
• Trust building wording describing our shopping process
• Adding an explicit guarantee that people can download the software lifelong

Note that we did not add any new feature nor the guarantee was new. It was just not described on the old page.

Question: What results did you get? Were you surprised by the results?
Answer:  Improvements started with about 50% but settled on about 35-40% after some time. I was very happy with the result but didn’t trust it. What was measured by VWO were the clicks into the shopping cart, not the actual sales.

So I measured the sales with the funnel visualization of Google Analytics which showed an increase of 17.7% of actual sales. It also proved that the number of clicks into the shopping cart were about the same what VWO had measured. It shows that the bounce rate within the shopping cart was somewhat higher in the winning combination but the overall increased click rate outtakes the bounce rate.

In the second phase, we removed some whitespace and moved the product name above the box shots and gained another 10% increase in clicks and the final increase in sales settled at about 20%.

Question: Any lessons which can be derived from your test?
Answer:  Don’t think your website is good as it is. Always test, always improve. It’s a slow process but worth doing.

Our concrete test shows that a clear modern design improve sales. Especially in the sales process trust building is very important.

Question: How valuable was Visual Website Optimizer for this test?
Answer: Is was so incredible easy to create the test (A/B split test in this case) that it takes nearly no time. Because we use an external shopping cart we can not measure in the cart itself so the feature of VWO tracking a click on a link without loading the page made it possible to run the test at all.

5 Comments »

A picture is worth a thousand words. And a video is worth laboring through hundreds of documentation pages and forum threads. The following video shows how to create a multivariate test using Visual Website Optimizer in well under 4 minutes.  The short tutorial is a must watch if you have heard about multivariate testing in articles/blogs; and always wondered what exactly is it. Even if you haven’t heard about multivariate testing, don’t worry – this tutorial with demystify that. Basically, multivariate testing is a technique where you vary several elements on your page to see which version results in maximum conversions (that is, form submissions in this case).  A successful multivariate test can often result in 50-100% increase in conversions (case study to come out soon).

Without any further blabbering, here is your video tutorial on multivariate testing (click Play to start the video):

1 Comment »