One of the questions we often hear from users who are new to A/B testing is what should they test. This does seem like a simple question but it can be quite baffling to the new-comers. This post is one such attempt to narrow down certain options to test, the changes that worked for others might work for you. The change here that we are talking about is testing having a video on homepage or product page. Previous case studies prove that it usually works wonderfully. One such case study is video increases conversion rate of a landing page by 46% and in another one they tested two call to actions ‘Watch the video’ v/s ‘Get Instant Access’ and found out that the video one increased newsletter signups by 28%.
In this latest case study, one of our clients Buy Real Twitter Followers made a variation with a small video explaining about their service on the homepage. This little change helped them increase their sales by 216%. Buy Real Twitter Followers was formed as a response to the need made by the growing Twitter advertising market. Getting Twitter followers has been a large concern by many network marketers and businesses whose primary aim is to advertise through Twitter.
Visitors usually tend to skip the blocks of text. This case study only goes on to prove that users much rather prefer to see the video for the product validation than reading block of text to get the gist of the product.
Original Page
Variation Page
This is what Rene Harvey had to say about the A/B testing tool they used: Visual Website Optimizer: “VWO was very helpful, it was very easy to use and provides a room for designers such as myself to customize without having to change the original code.”
As always, everything is open to testing and the idea is to find what change benefits the most. Let us know what you think about this case study in the comments below.
Hyundai, the Korean giant, not just makes well-engineered cars, they have well-engineered online strategy as well. In the Dutch market, they hired one our certified partners, Traffic4U to optimize conversion rate on their lead-generation pages. Traffic4u is an experienced, international online marketing agency, specializing in Result Driven Online Marketing and since they are our partners, they chose Visual Website optimizer for testing and optimization on Hyundai.nl
In this case study, we catch up with Janco Klijnstra from Traffic4U who talks about how they used Visual Website Optimizer for increasing conversion rate by 62% for Hyundai.
Hyundai has landing pages for all of their car models where people can request for a test drive or download a brochure. These landing pages mainly get traffic from paid advertising campaigns (but get some direct and SEO traffic too). They wanted to make most from this traffic hence partnered with Traffic4U for conducting a multivariate test on all car model pages.
There were several goals for this test that they wanted to optimize:
The main difference between A/B testing and multivariate testing is that in A/B testing variety of changes are done in a single variation while in multivariate testing, every change you make creates a new variation to be tested. So, multivariate testing is useful if you have multiple ideas to test on a page as it tells you exactly which changes impact your conversion rates and which don’t.
Hyundai’s car landing pages have a lot of different elements (car headline, car visuals, description, testimonials, etc.) so it was essential to setup a multivariate test in order to understand which elements influence a visitor’s decision to ask for a test drive or download a brochure.
Traffic4U setup a multivariate test and decided to create variations of following sections of the page:
A total of 8 combinations (3 sections, 2 variations each = 2*2*2) were generated for this multivariate test. Here’s screenshot of the original page:
- Original page -
The results of this multivariate test were phenomenal. One of those variations increased conversion rate (request for test drive or brochure) by 62%. And there was a staggering 208% increase in clickthrough rate (step 1 to step 2). Out of the total 8 combinations, can you guess which one increased conversions?
Well, the combination with SEO text, extra buttons and larger images did the wonder. It’s amazing how Traffic4U was spot on for all three changes. Here’s how the variation looked like:
- Winning variation (62% increase in leads) -
The results of this multivariate test are so phenomenal that it won a silver award in annual WhichTestWon awards!
Traffic4U used lot of heuristics and best practices experience for coming up with variations for this test (larger pictures, clear call to action buttons, etc.). This test demonstrates that doing a proper multivariate test can validate those best practices. It may have been certainly possible that adding SEO text decreased conversions at the expense of extra traffic, however setting up a multivariate or A/B test can settle that dilemma easily.
When we asked Traffic4U about their comments on Visual Website Optimizer, here’s what they had to say:
Visual Website Optimizer was very valuable it made it possible for us as an agency to build our own variations, test on a group of pages and measures different goals at once. Also the Analytics plug-in was very helpful in the analysis for segmentation purposes.
This case study is an excellent example of how a large company (such as Hyundai) works with a specialized conversion rate optimization agency like Traffic4U to use multivariate testing for increasing business metrics such as test drive requests and brochure requests. If it worked for Hyundai, it can work for you too!
Signup for a free 30 day trial account of Visual Website Optimizer to easily setup multivariate tests on your website to increase sales and conversions.
In the past, we have featured a number of A/B testing case studies. In many of those case studies, a number of elements change which contribute to increased sales and conversions. However, this case study is special. Here a single (seemingly trivial) change was responsible for increasing sales by as much as 34%. This is unprecedented and, of course, is an amazing story which demonstrates the power of A/B testing.
That’s why I chatted with Chris Muktar of WikiJob.co.uk, UK’s largest graduate jobs website. Started in 2007, it now attracts over 500,000 visits a month. Started by two friends, it has grown to be a profitable company employing several staff, and proudly boasts many of the world’s largest brands as their clients.
WikiJob use Visual Website Optimizer for A/B testing and below is the interview where Chris talks about how they managed to boost bottomline sales by 34% just by testing a small element.
What conversion goals were measured in the test?
There were two conversion goals. The first was to click through to Paypal checkout. The other was to complete it and actually buy something.
On which page did you run the test?
http://www.wikijob.co.uk/aptitude-tests
What is the traffic source? Organic, direct, PPC, etc.?
Most of the traffic was from our main http://www.WikiJob.co.uk website. The traffic to our website is almost entirely organic.
Which part of page did you select for the test and what variations did you test?
We’ve tested a number of things in the past and had some great results. This time we were testing the effect of three lines of testimonials given by customers and their impact on sales. Below are screenshots.
- Original Page - 
- Variation (with screenshots): 34% increase in sales - 
Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
A previous test displayed the same testimonials further down the page and had no discernable impact on conversion. However, it’s commonly thought that social proof can help in the sales process, so we wanted to leave nothing to chance and test it.
What results did you get? Were you surprised by the results?
Testimonials increased sales by 34%. The testimonials we used are very ‘sober’ (compared with the overly enthusiastic ones you so often see in marketing literature). The test results were surprising. Although such increases of sales can be quite normal in split testing, I did not think that testimonials would make such a difference (and indeed put off testing them, thinking they were irrelevant). The increase in revenue was very substantial.
Any lessons which can be derived from your test?
Social proof is important, and it is important to test everything. I may try putting some small pictures next to each testimonial to see if this improves things. [Editor's note: do that! Our previous split tests show that human pictures can potentially double conversion rates]
How valuable was Visual Website Optimizer for this test?
Visual Website Optimizer has been invaluable. You pay for the service once, but the increase profits and revenues continue into perpetuity. Through the use of several iterations of tests, we have increased sales threefold.
Last week we ran an A/B test contest where you had to guess which button increased newsletter signups by 28%. Was it ‘watch a video’ or was it ‘get instant access‘? We got a total of 51 submissions (tweets + blog comments) and would you believe it: 51% said ‘Get Instant Access‘ would win while 49% said ‘Watch a video‘ will win. This is incredible and clearly demonstrates how hard is it to guess what is going to work on your website!
Before announcing the correct answer, we’d like to announce the winner of $3000 worth Visual Website Optimizer subscription. Please join me for the drum roll… and the winner of this contest is: Stephan Schubert and here is the winning tweet. Congratulations, Stephan!
I recently interviewed The Social Man (one of the Visual Website Optimizer customers) on their successful split test. As you will read, this test is typical example of how a single change (in call-to-action) can result in significant increase in conversions. Almost all our case studies make the same point about A/B testing which is worth repeating twice: (seemingly) small changes can increase conversions on a landing page. So, you should always be testing.
Here is the full interview:
1. What was the conversion goal of the split test?
To convert visitors to email newsletter subscribers, and to get them to see a video about how a guy can learn to “talk like a ten”.
2. On which page did you run the test?
http://www.becomeunbreakable.com/1/talk-like-a-ten/indexVWO.php
(Please note – this is a different page than the original, but keeps all the same content. We track lead sources and use different landing pages for traffic from different sources)
3. What is the traffic source? Organic, direct, PPC, etc.?
Cold traffic from a dating website – CPM. We tested a lot of different bids and were very surprised about the most profitable bid price.
4. Which part of page did you select for the test and what variations did you test?
We initially ran a test with three different variations each of headline and sub-head. Once we found the best combination for those, we split the “Submit” button on a whim. One said “Free Instant Access” and one said “Watch The Video”. See images below:
Version A: Get Instant Access (11.9% conversion rate)
Version B: Watch the Video (15.3% conversion rate)
Editor’s note: so now you know that ‘Watch the video‘ increased conversions from 11.9% to 15.3%. A total increase of 28%!
5. Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Pushing for the action of “Watch The Video”. The subhead that won (“Get The Free ‘Talk Like A Ten’ Video Now!!!”) also mentioned that they could “get” the “video”.
6. What results did you get? Were you surprised by the results?
15% opt-ins for cold traffic off this particular site is amazing – We’re running ads/offers into a few very nichey demographics on that site that generate huge CTR’s (.300-.400; in comparison, the CTR on this campaign is .086 for our best creative) and haven’t beaten this conversion for any other squeeze page.
7. Any lessons which can be derived from your test?
I’m reminded of the parable where an old friend of Henry Ford’s said “Henry, why don’t you ever buy any bolts from me?” to which Mr. Ford replied “Heck Joe, you’ve never asked!” It’s always worth including action verbs like “get”, “watch”, “download”, “enter” etc. in tests.
8. How valuable was Visual Website Optimizer for this test?
It’s so much easier – and more powerful – than Google’s website optimizer. We’re finding it to be an indispensable tool that made me excited about running tests again.
9. Short background information about your business.
The Social Man is in business is to make men awesome. We have a variety of products and coaching packages that cover everything from flirting over text messaging, to how to be sexier, to overcoming social anxiety. I previously ran an enterprise software company, and feel very blessed to be doing something I love that’s employs passionate people and has helped so many men. My business, Turnseven, Inc., manage a few other niche brands in addition to The Social Man.
I just published a guest post on Smashing Magazine titled Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates.
Essentially, there are five steps to increasing conversion rate:
If these steps sound complicated to you, I recommend you to read the extensive tutorial which has numerous examples.
My article explains multivariate testing by means of a case study where I tested following variations on a software download page (notice color and text changes):
Can you guess which variation produced maximum downloads? Well, the end result of this test was that #10 combination (in the screenshot above, one with ‘Download for Free’ in red) had 60% improvement in conversion rate. That’s the power of multivariate testing.
Read the full case study and tutorial: Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates. I hope you like it!
Many times A/B testing is not limited to a single conversion goal. In fact, your test variations usually affect many different conversion goals on your site such as free trial signups, paid signups, newsletter subscription, etc. Measuring all these goals for a test is important because a variation may work brilliantly for one goal (e.g. it increases free signups) but may perform worse for your other goals (e.g. decreases your paid signups). If you don’t measure multiple goals for your A/B test, you are essentially flying blind in the dark and may end up making wrong decisions (based on a single conversion goal).
Case Study
A brilliant example of this is a recent A/B test by one of our customers: Guido Jansen (a Magento specialist). He runs Dutchento.org which is the official Dutch community for Magento CMS. The goal of his recent A/B test was to increase subscriptions for Dutchento’s newsletter and RSS subscription. The call-to-action for subscription is located in a box on all pages of the site. See the control version of subscription box below and note that there is no explicit incentive for a visitor to subscribe:
Guido tested a variation of this subscription box which included a title and some benefits in a bullet list. Here is his hypothesis for increasing subscriptions:
People don’t just subscribe to a newsletter or newsfeed for nothing, you should convince them it has added value above just visiting the website. So what I wanted to test is if adding convincing reasons to subscribe would increase the newsletter and newsfeed subscription rate. I measured the impact of the convincing reasons on clicks on both the newsletter and newsfeed links.
Here is how his variation (of subscription box) looks like:
Results
As expected, Guido saw a significant improvement of the newsletter click rate (+190.31%). However, newsfeed click rate decreased (-44.46%) which did surprise him (and us!). He expected that the convincing reasons would affect both positively, but apparently it had a negative effect on newsfeed clicks.
The reason why clicks on newsfeed decreased is not clear but we believe that the benefits in variation were so compelling that visitors chose to get the blog updates via email (where they will be sure to read them) rather than RSS reader (where they may miss them). A great way to get more insight into this would be to randomize the position of newsfeed / newsbrief to eliminate the positional effect of those links.
Testimonial
Guido used Visual Website Optimizer for A/B testing and here is what he had to say:
Visual Website Optimizer was very valuable [for testing]. It’s the easiest A/B and Multivariate testing tool I know. It’s great not to be dependent on the development department to create and run your tests.
Key Lesson
Whatever be the actual reason of decreased clicks on RSS feed, one key lesson is jumps out of the case study: always measure multiple conversion goals in your test. Relying on a single conversion goal hides a lot of valuable information from you. So, make sure you add multiple goals to your next A/B test.
Do you have more examples of using multiple goals in A/B test? What are your thoughts on this case study? Please let us know by leaving a comment below – will be happy to discuss!
Traffic coming to a specific blog post rarely goes on to read your other blog posts. A visit to homepage or product page from a blog reader is even rarer. No wonder, blogs have unusually high bounce rates. The Visual Website Optimizer blog gets a ton of traffic, but only a trickle of it ultimately converts into signing up for the free 30 day trial.
Our original blog design is shown below; note that we had a huge ‘I love Split Testing’ heading towards the top of the page. Even though that’s the name of the blog and ideally it should be retained, our hypothesis was that it may actually be acting as an attention barrier. Note that it competes for attention with the top navigation bar which has product logo and other links related to Visual Website Optimizer.
Original blog design (with header) -
Enter ‘Anti A/B testing‘ – removing the top header
We decided to do a simple anti A/B test to increase signups from the visitors (like you) who have come to read articles on this blog. Now, wait, you may ask what exactly an anti A/B test is. Well, it’s nothing but a term we coined to describe a test where you test presence of an element v/s absence of it. In a typical A/B test, you create variations of an element (say headline, call-to-action buttons, images, etc.). However, in an anti A/B test you remove the element altogether and then see the impact on conversions. In our anti-A/B test, we removed the blog header completely. See below how the blog now looks like without the header:
Removed header from blog: +60% increase in signups
Test Results
Guess what? The variation which did not have the blog header increased signups by 60%. And not just the signups, just about every conversion goal that we tracked showed a dramatic improvement. Have a look at the results below:
| Conversion Goal | Increase in conversion rate |
| Signups for free trial | +60% |
| Engagement rate (inverse of bounce rate) | +20% |
| Visit to signup page | +30% |
| Visit to homepage | +13% |
| Visit to blog main page | +78% |
As you can see, even visits to the main blog page increased by 78% for the variation that did not have the blog header. The overall positive results obtained from the test re-enforced our belief that even small changes on the pages can have dramatic results. (See other A/B and multivariate test case studies for more examples)
Which anti A/B test will you do?
So, next time you think about optimizing one of your pages, try to think about which element on page you can afford to remove. Perhaps it’s the large product screenshot or video that’s hurting the conversions? Or, perhaps it is the trust / secure logo that’s decreasing the conversions?
It has been criticized, but it is always guaranteed to work. What is that we are talking about? Yes, you guessed it right: red buttons! No matter how many people consider such A/B tests as a trivial exercise, every now and then they have been demonstrated to increase conversions.
The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.
All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!
Here are different variations that were tested:
| With text buttons | With green buttons | With red buttons |
 |
 |
 |
The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)
As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.
The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.
In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.
ReplaceDirect, a Visual Website Optimizer customer, is a successful Dutch e-commerce site. It’s one of the biggest retailers in the Netherlands in the field of parts and accessories for notebooks and other mobile equipment. The main focus is on energy products like batteries and power supplies, but the product range is quite extensive.
They recently did an A/B test using Visual Website Optimizer where the winning variation reduced cart abandonment by 25%. For e-commerce sites, cart abandonment is one of the most frustrating aspects of business. After all, there’s no company that likes a potential customer to abandon the purchase after spending tons of time on site research and selecting the right products. Fortunately, cart abandonment can be fixed using A/B testing and this case study demonstrates how ReplaceDirect did it.
A recent study by Forrester found out that shipping costs rank as the number one reason for shopping cart abandonment. Combined with other reasons found out in the research, it can be concluded that visitors abandon their shopping cart because they are unsure about additional costs or surprised to find out (only after entering the payment process) that they are actually charged more than they expected. In order to prevent such surprises, ReplaceDirect already tells the customer in an early stage that no shipping costs will be added. So the customer is clearly informed right from the start. This is exactly what ReplaceDirect wanted to test: will it help reduce cart abandonment when a clear order overview is provided during the initial steps of checkout?
Another cart abandonment study by Paypal
They tested their second step of the checkout procedure. It’s the page where customers are asked to fill out their personal information and shipping address. ReplaceDirect made several changes to the page. Probably the most important change was the insertion of an order overview, comprising the product(s), total costs and delivery date. ReplaceDirect also mentions several benefits at this page; these were changed to more relevant benefits specifically matching this step in the order process. Finally the layouts of the page and the form were changed to yield a cleaner look. Several fields, which were not absolutely necessary, were removed. Following are the screenshots:
Control page
Variation page: 25% reduction in cart abandonment
Their new version performed significantly better, reducing cart abandonments by 25%. As a direct result of this reduction, they saw a 12% increase in sales (14% by the time of pushing all new traffic to the new page). ReplaceDirect says they expected the new version to perform better, but an increase that high turned out to be a pleasant surprise. Well, it is always nice to have a 14% increase in sales without releasing any new products or without spending money on ways to get more traffic to the website. 
When we asked them if they had any lessons for other e-commerce sites; here is what they had to say:
Try to give the customers exactly the information they need at that particular page or section and leave out the redundant. Always put yourself in the customer’s shoes. This may not be easy so: test it!
ReplaceDirect also had some praise for Visual Website Optimizer:
VWO was very valuable. The ease to set up a test and real-time results made it great to work with.
We are very happy that ReplaceDirect reported these great results by doing an A/B test using our tool.
If you manage an e-commerce site without doing any A/B split testing, you are certainly leaving a lot of money on the table.
We had launched clickmaps and heatmaps for A/B tests recently in Visual Website Optimizer. Since then it has become the most popular feature amongst our users. We aren’t surprised by the success of this new feature because it gives a new angle to A/B test results. While you can always get information about conversion rate, bounce rate and other hard metrics from standard reports, visualizing “hotspots” on a page has a charm of its own.
Based on feedback from a user, we rolled out a new minor feature: consolidated clickmaps. Unlike traditional clickmaps, which show results only for a particular page, this new feature will aggregate click activity on ALL your website pages to show which parts get most clicks. This is useful if you have a website that has common template/theme across different pages (e.g. header, footer, sidebar, product box, etc.). Using consolidated clickmaps, you will be able to see statistics for elements common to your website.
This feature came handy when I analyzed visitor activity on my personal blog. All I had to do was to enter a URL pattern (http://www.paraschopra.com/blog*) which instructed VWO to generate a consolidated heatmap for all blog posts (technical detail: VWO combined heatmap data for all pages matching the URL pattern; * matches anything and that’s why the entered URL pattern matched all blog posts).
Here is the consolidated heatmap for all my blog posts:
Contrast this with a heatmap only for a single post:
As you can see, the consolidated heatmap has much more information than a single blog post heatmap. That is the real utility; you get to observe patterns of visitor behavior across your website which you may have otherwise missed or would require you browsing through tens of different heatmaps and then stitching patterns in your head.
Using this feature, you also get to see consolidated clickmaps. Here is a screenshot:
The consolidated clickmap clearly shows that 10% of all clicks on the page occur on Visual Website Optimizer banner on my blog. Some more statistics from the clickmap:
The stats are bit on the lower side but if you observe consolidated heatmap, you will see a lot of click activity on post text. This means that even though visitors aren’t clicking on links, they are still interacting with the post and reading it!
I hope you like the new feature! If you’d like to give it a spin simply login to your account (it is enabled by default for all accounts). Or, if you aren’t an existing user, signup for a 30 day (no-obligations) free trial here.
PS: Technical details. If you are wondering how we are able to generate consolidated heatmap, here is the trick: we don’t store absolute X and Y positions of visitor clicks. Rather we store xpath of the element being clicked. Common elements on the website have same xpath across different pages, hence we are easily able to aggregate that data.
