Beamax, a Belgium based company, manufactures and distributes projection screens for home cinemas and meeting rooms world-wide. They wanted visitors on homepage to go to a site dedicated to ex-demo and one-off items that are sold directly to consumers. (All other screens are sold indirectly through resellers). They admit that it’s a bit odd to drive away visitors to another site from your main page, but they wanted to clear up some space in the warehouse that was taken up by uncommon items.
So, they decided to do a simple A/B test using Visual Website Optimizer. Just above the product images on homepage, they put a standard link promoting the other website. It said:
Great deals on brand new and ex-demo screens here
To increase clickthroughs on link, they tested a red link (with same text) because they felt it would out-perform the standard blue that they use. Plus, it’s something direct marketers use in “real” mail pieces too. As another variation, they transformed link into something banner-like that they thought would even have more impact. Their hypothesis was that the banner version the sure-fire winner. See screenshots below:
Original page
Page with red link
Page with red banner
So, any guesses on which version got maximum clicks: blue, red or banner version? Well, the red link and banner both outperformed the blue link and that wasn’t a surprise. But the eye-opening result was that the red link winning from the banner. The improvement of red link compared to the original blue link was pretty big too: 53.13%. (Note we have some other case studies online which demonstrate how red link outperforms the default link. Here are two examples: PDFProducer case study and GSM.nl case study)
Otto Tromm, CEO of Beamax, stresses the importance of waiting for statistically significant results. He says:
In the early stage of the test, the banner was the big winner. But, over time (when the results got more reliable), the red link outperformed the banner. That taught me not to jump to conclusions.
And it was tempting to declare an early winner, because initial results proved my gut feeling. The test proved me wrong, so it teaches you to stay humble too.
So would I implement a red link vs a banner blindly next time? No, I would test it!
Choice of the right tool is certainly very important when are you are doing A/B tests. Beamax chose Visual Website Optimizer for the job (just like thousands of other businesses). Here’s what Otto from Beamax has to say about the tool:
I am not a designer or coder and we use Mod-x and CMS defined templates for nearly all pages. So I neither want to call on experts for every test I do, nor do I want to mess up their work. Visual Website Optimizer made it easy for a non-tech guy to do the tests and keep our designer and programmer focused on their own projects.
With Google’s solution, it was a lot more work to implement tests, which is why I stopped using it. Just couldn’t get it all done myself, which is important when you have an idea and quickly want it implemented.
Hope you liked this case study! If you have any comments or suggestions, we are all ears.
Recently on Hacker News, someone commented that A/B testing has become like snake oil, making grand promises to be a panacea for increasing conversion rates. This comment troubled me not because the commenter was wrong (he was!) but because how certain people view A/B testing. Let me put this straight and clear:
In fact, we recently featured a guest post by Noah of Appsumo who revealed that only 1 out of 8 split tests that they run produce any actionable results. Yep, read that once again: only 1 out of 8 split tests work. Snake oil guarantees to be a cure for all diseases; A/B testing has no such guarantees.
I think the reason why some people would view split testing suspiciously is because we release a lot of A/B testing case studies. In fact, with a user base so large (6000+ at the time of writing), we see many of our customers getting good results all the time and they if want to share those results with the world, we publish them on our blog.
So, when one week we publish a case study titled 3 dead-simple changes increased sales by 15% and the next week we publish another case study titled How WriteWork.com increased sales by 50%, it naturally makes some people assume using Visual Website Optimizer to do A/B testing would ensure a growth in sales. No, using Visual Website Optimizer won’t guarantee anything. It is just an A/B testing tool (albeit, a very good one!) but the real work done is always by the craftsman, not the tool s/he uses.
Most success stories or case studies (be it about A/B testing, getting rich or losing weight) that you read about on the Internet suffer from something known in psychology as survivorship bias. In context of A/B testing, what it means is simple:
If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.
Obviously, 99 case studies about how A/B testing did not produce any results in unexciting and will bore our blog readers. Hence, this post is a short reminder about all those case studies that don’t produce results.
The reason we publish A/B testing case studies is to show potential of what can be achieved using A/B testing. It is similar to holding an art exhibition for Picasso paintings which inspires people to appreciate the work and perhaps pick up brush themselves to start painting hoping to achieve level of success that Picasso achieved. To reiterate, just as picking up a brush and canvas does not guarantee good art, doing A/B testing does not guarantee increase in sales.
In fact, specifically to A/B tests, replicating them almost never works. So, if you read a case study of how video increases conversions by 46% and then implement it on your site hoping for the same magical increase, it may or may not happen. What worked for them may not work for you! So, why publish case studies at all? Here’s why:
A/B testing case studies give ideas on what you can test on your website
Yep, that’s all there is to A/B testing case studies.
And, yes, I hope I convinced you that A/B testing is not snake oil!
It has been criticized, but it is always guaranteed to work. What is that we are talking about? Yes, you guessed it right: red buttons! No matter how many people consider such A/B tests as a trivial exercise, every now and then they have been demonstrated to increase conversions.
The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.
All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!
Here are different variations that were tested:
| With text buttons | With green buttons | With red buttons |
 |
 |
 |
The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)
As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.
The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.
In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.
