Visual Website Optimizer

This case study was first published by VisitNorway.org, which is one of our customers. They have shared their experience of performing A/B tests on their website. The original blog post was in Norwegian and we translated it into English and are reproducing the same below.

We are constantly striving to improve Visitnorway.com, and we have generally used traditional tools for analytics as well as usability tests and surveys to assess needs of our visitors and accordingly customize the content and functionality.

In January, we began using a new A/B testing tool [Visual Website Optimizer] for quickly and effectively measuring the impact of changes in text, design and functionality.

Based on input from users, experts or hypotheses, when we decided to conduct a test, we set up the variations that we wanted to test against the original. Based on the results of tests, this helped us decide whether we want to implement the variation on website permanently or not.

For example, it is very easy to test the title that gives the most clicks in an article. We just need to click on the title and change it, see the screenshot below:

Results from some of the tests we have conducted

1) Click to destination company websites from landing pages

We wanted to find out about simple design change on landing pages to increase the number of clicks to the destination’s websites. At first, we only increased the size of the link text and converted it into bold.

The test was done on VisitOSLO pages of visitnorway.com. We got a positive increase of 70 – 75% in click through rate simply by changing the size and have stronger text, see the screenshot below.

We therefore proceeded with this change on the landing page and increased visits to destination company website.

2) What should be the “Booking” name called in the top menu?

We tested different names for the menu item “Booking” in English, Norwegian and Spanish, hoping to lead even more of those who were interested in booking a holiday into the booking section .

For English, the text “Booking” won with an improvement of 39% against the “Book Online” as it originally stood. “Online booking” had a 14% improvement to the original.

The Norwegian word for “Order” with an improvement of 114% (!) won against the original “Book travel”. See screenshot below for an overview of the other variants.

In the Spanish menu, we wanted to see if it was best to write the “Booking” in Spanish or English. Since the Book Norway does not yet have Spanish version, we were curious about the bounce rate.

With regards to how many people went to booking section, “Booking” had a decrease of 48% in English (Variation 1) in relation to “Reservas” (Control). See screenshot below.

Bounce rate was similar for both of them, so we eventually used the “Reserve” in the Spanish menu.

Based on these results, we changed the text in the menu to the “winners” of these three tests.

3) Moving the “Order” link in the main menu for the Norwegian edition

We wanted to see how much change there was in the number of clicks to the Book Norway when we moved the “Order” from the second position to the second last and last position in the main menu.

We had a decline of 72-73% in the number of clicks, see the screenshot below.

So as the numbers speak, we did not move not the “Order” link.

Conclusion and Lessons

A/B testing is an important method for us to determine whether it is wise to move forward with new concepts or changes, before we spend a lot of time and money on design and development. We will use it actively to get real decision-making data.

Our findings show that one must be careful of what you call “booking” in the various languages ​​- words are tremendously powerful.

Finally, we chose Visual Website Optimizer because it was easy to use and had enough functionality to allow us to carry out the tests that we want.

Happy testing!

8 Comments »

Beamax, a Belgium based company, manufactures and distributes projection screens for home cinemas and meeting rooms world-wide. They wanted visitors on homepage to go to a site dedicated to ex-demo and one-off items that are sold directly to consumers. (All other screens are sold indirectly through resellers). They admit that it’s a bit odd to drive away visitors to another site from your main page, but they wanted to clear up some space in the warehouse that was taken up by uncommon items.

A/B testing link colors

So, they decided to do a simple A/B test using Visual Website Optimizer. Just above the product images on homepage, they put a standard link promoting the other website. It said:

Great deals on brand new and ex-demo screens here

To increase clickthroughs on link, they tested a red link (with same text) because they felt it would out-perform the standard blue that they use. Plus, it’s something direct marketers use in “real” mail pieces too. As another variation, they transformed link into something banner-like that they thought would even have more impact. Their hypothesis was that the banner version the sure-fire winner. See screenshots below:

Original page

Page with red link

Page with red banner

What type of link got most clicks?

So, any guesses on which version got maximum clicks: blue, red or banner version? Well, the red link and banner both outperformed the blue link and that wasn’t a surprise. But the eye-opening result was that the red link winning from the banner. The improvement of red link compared to the original blue link was pretty big too: 53.13%. (Note we have some other case studies online which demonstrate how red link outperforms the default link. Here are two examples: PDFProducer case study and GSM.nl case study)

Lessons learned: patience pays

Otto Tromm, CEO of Beamax, stresses the importance of waiting for statistically significant results. He says:

In the early stage of the test, the banner was the big winner. But, over time (when the results got more reliable), the red link outperformed the banner. That taught me not to jump to conclusions.

And it was tempting to declare an early winner, because initial results proved my gut feeling. The test proved me wrong, so it teaches you to stay humble too.

So would I implement a red link vs a banner blindly next time? No, I would test it!

Visual Website Optimizer: how important was it?

Choice of the right tool is certainly very important when are you are doing A/B tests. Beamax chose Visual Website Optimizer for the job (just like thousands of other businesses). Here’s what Otto from Beamax has to say about the tool:

I am not a designer or coder and we use Mod-x and CMS defined templates for nearly all pages. So I neither want to call on experts for every test I do, nor do I want to mess up their work. Visual Website Optimizer made it easy for a non-tech guy to do the tests and keep our designer and programmer focused on their own projects.

With Google’s solution, it was a lot more work to implement tests, which is why I stopped using it. Just couldn’t get it all done myself, which is important when you have an idea and quickly want it implemented.

Hope you liked this case study! If you have any comments or suggestions, we are all ears.

7 Comments »

Recently on Hacker News, someone commented that A/B testing has become like snake oil, making grand promises to be a panacea for increasing conversion rates. This comment troubled me not because the commenter was wrong (he was!) but because how certain people view A/B testing. Let me put this straight and clear:

A/B testing does NOT guarantee increase in sales and conversions

In fact, we recently featured a guest post by Noah of Appsumo who revealed that only 1 out of 8 split tests that they run produce any actionable results. Yep, read that once again: only 1 out of 8 split tests work. Snake oil guarantees to be a cure for all diseases; A/B testing has no such guarantees.

I think the reason why some people would view split testing suspiciously is because we release a lot of A/B testing case studies. In fact, with a user base so large (6000+ at the time of writing), we see many of our customers getting good results all the time and they if want to share those results with the world, we publish them on our blog.

So, when one week we publish a case study titled 3 dead-simple changes increased sales by 15% and the next week we publish another case study titled How WriteWork.com increased sales by 50%, it naturally makes some people assume using Visual Website Optimizer to do A/B testing would ensure a growth in sales. No, using Visual Website Optimizer won’t guarantee anything. It is just an A/B testing tool (albeit, a very good one!) but the real work done is always by the craftsman, not the tool s/he uses.

Survivorship Bias: it helps to know a thing or two about it

Most success stories or case studies (be it about A/B testing, getting rich or losing weight) that you read about on the Internet suffer from something known in psychology as survivorship bias. In context of A/B testing, what it means is simple:

If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.

Obviously, 99 case studies about how A/B testing did not produce any results in unexciting and will bore our blog readers. Hence, this post is a short reminder about all those case studies that don’t produce results.

Why publish A/B testing case studies at all

The reason we publish A/B testing case studies is to show potential of what can be achieved using A/B testing. It is similar to holding an art exhibition for Picasso paintings which inspires people to appreciate the work and perhaps pick up brush themselves to start painting hoping to achieve level of success that Picasso achieved. To reiterate, just as picking up a brush and canvas does not guarantee good art, doing A/B testing does not guarantee increase in sales.

In fact, specifically to A/B tests, replicating them almost never works. So, if you read a case study of how video increases conversions by 46% and then implement it on your site hoping for the same magical increase, it may or may not happen. What worked for them may not work for you! So, why publish case studies at all? Here’s why:

A/B testing case studies give ideas on what you can test on your website

Yep, that’s all there is to A/B testing case studies.

And, yes, I hope I convinced you that A/B testing is not snake oil!

1 Comment »

It has been criticized, but it is always guaranteed to work. What is that we are talking about? Yes, you guessed it right: red buttons! No matter how many people consider such A/B tests as a trivial exercise, every now and then they have been demonstrated to increase conversions.

The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.

All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!

Here are different variations that were tested:

With text buttons	With green buttons	With red buttons

The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)

As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.

The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.

In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.

3 Comments »