Tulos, a Finnish SEO and conversion rate optimization agency and a customer of Visual Website Optimizer did some A/B testing on its client Scandinavian Outdoor Store (an online store that sells high class outdoor clothing and equipment). Following is an interview with Tulos, where they detail just how they managed to increase conversions by 127% by doing a simple headline change.
Q: Which page did you run the test?
A: The front page of the product category for men’s clothing.
Q: What was the conversion goal of the test?
A: The conversion goal was to get more visitors to add products into the shopping cart and to have fewer bouncing visitors from the test page.
Q: Which part of page did you select for the test and what variations did you test?
A: We tested the variations for the headline of the page. The original headline only included the text – “Men’s Clothing”. The winning test combination included the following test (free transcription from Finnish to English) “Order Men’s Clothing easily for bargain prices”.
The original version didn’t have any text under the headline. We tried adding a short block of text that included a few links to best bargain products of this product category.
Q: Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
A: We tested how the mention of bargain prices affected the bounce rate and engagement to add products to the shopping cart. The original version did not have any text about the bargain prices on the front page of this product category so it was quite obvious that this test variation would have a great impact on the performance.
Q: What results did you get? Were you surprised by the results?
A: The variation that included more selling headline and bargain text outperformed the original version by having a 127% improvement against the original plain version.
Q: Any lessons which can be derived from your test?
A: You shouldn’t underestimate or rule out the value of great text content on product category pages of online stores.
Q: How valuable was Visual Website Optimizer for this test?
A: VWO was a great and easy to use tool for setting up the A/B test without any technical installation issues.
Many times A/B testing is not limited to a single conversion goal. In fact, your test variations usually affect many different conversion goals on your site such as free trial signups, paid signups, newsletter subscription, etc. Measuring all these goals for a test is important because a variation may work brilliantly for one goal (e.g. it increases free signups) but may perform worse for your other goals (e.g. decreases your paid signups). If you don’t measure multiple goals for your A/B test, you are essentially flying blind in the dark and may end up making wrong decisions (based on a single conversion goal).
Case Study
A brilliant example of this is a recent A/B test by one of our customers: Guido Jansen (a Magento specialist). He runs Dutchento.org which is the official Dutch community for Magento CMS. The goal of his recent A/B test was to increase subscriptions for Dutchento’s newsletter and RSS subscription. The call-to-action for subscription is located in a box on all pages of the site. See the control version of subscription box below and note that there is no explicit incentive for a visitor to subscribe:
Guido tested a variation of this subscription box which included a title and some benefits in a bullet list. Here is his hypothesis for increasing subscriptions:
People don’t just subscribe to a newsletter or newsfeed for nothing, you should convince them it has added value above just visiting the website. So what I wanted to test is if adding convincing reasons to subscribe would increase the newsletter and newsfeed subscription rate. I measured the impact of the convincing reasons on clicks on both the newsletter and newsfeed links.
Here is how his variation (of subscription box) looks like:
Results
As expected, Guido saw a significant improvement of the newsletter click rate (+190.31%). However, newsfeed click rate decreased (-44.46%) which did surprise him (and us!). He expected that the convincing reasons would affect both positively, but apparently it had a negative effect on newsfeed clicks.
The reason why clicks on newsfeed decreased is not clear but we believe that the benefits in variation were so compelling that visitors chose to get the blog updates via email (where they will be sure to read them) rather than RSS reader (where they may miss them). A great way to get more insight into this would be to randomize the position of newsfeed / newsbrief to eliminate the positional effect of those links.
Testimonial
Guido used Visual Website Optimizer for A/B testing and here is what he had to say:
Visual Website Optimizer was very valuable [for testing]. It’s the easiest A/B and Multivariate testing tool I know. It’s great not to be dependent on the development department to create and run your tests.
Key Lesson
Whatever be the actual reason of decreased clicks on RSS feed, one key lesson is jumps out of the case study: always measure multiple conversion goals in your test. Relying on a single conversion goal hides a lot of valuable information from you. So, make sure you add multiple goals to your next A/B test.
Do you have more examples of using multiple goals in A/B test? What are your thoughts on this case study? Please let us know by leaving a comment below – will be happy to discuss!
A lot of our potential customers are concerned whether A/B or multivariate testing will have an effect on their search engine rankings. They (obviously) aren’t looking to improve their rankings by using A/B testing, rather their concerns relate to the myth that testing may negatively impact search engine optimization efforts.
Broadly, there mainly two major concerns:
Why A/B testing is not content cloaking
Cloaking content was all rage back in the early days of search engines (think 1990s). In those days, SEO was all about keyword stuffing. So, smart SEO geeks used to display a page full of keywords when a search engine bot visited to crawl / index the page. While if a user visited the same page, they displayed the default (normal) version. This strategy of keyword stuffing used to work like wonders so naturally search engines started devising clever ways to detect and penalize such cloaking and it may appear that pages where A/B testing is on will also get penalized because it is kind of cloaking.
Good news is that thanks to Google’s PageRank algorithm, keyword stuffing no longer works. So, there is little incentive for search engines to penalize content cloaking. Moreover, unlike yesteryears’ static HTML sites, today’s search engines have come to expect highly dynamic AJAX driven sites. So, they no longer consider swapping content dynamically as cloaking. If you use a multivariate testing software to swap different parts of your page, it is not cloaking! Also, all the major search engines (Google, Yahoo, Bing, etc.) are well versed with concept of A/B split testing and they do many such tests on their own sites every single day. You wouldn’t expect search engines to penalize others for doing the same activity that they themselves do regularly, would you?
Key lesson: Doing A/B testing by dynamically swapping elements on page (using JavaScript) is NOT content cloaking. It is absolutely safe and should not have any negative impact on Search Engine Optimization efforts.
Split URL testing (redirecting traffic to multiple versions) and the issue of content duplication
As I mentioned above, this is another issue that worries many prospective customers. While one approach for split testing is to dynamically swap page elements (such as headlines, images, text, etc.), another approach is to host different variations of page on your website. So, if you want to split test product.html, you will create variation pages and host them as product1.html, product2.html, etc. A split testing software will redirect all traffic coming to product.html to the variation pages. Since product1.html and product2.html will have similar content as product.html, it worries some people that search engines will consider them duplicate content and penalize rankings.
An important point to note here is that search engines only penalize if you steal or host content from a different domain; here all variations pages host on your own site and you OWN that content and are free to do with it whatever you want to. Many dynamic websites (shopping carts, directories, etc.) today host the same content in different formats. Search engines don’t penalize them because that unique content is only found on their domain and no where else. Same is the case with your split testing URLs. You don’t get negatively impacted by it because you haven’t stolen that content — it is all yours!
A more important point is answer to this question: how will search engines come to know that there are different variations of your main page? Your variation pages can only be reached if there is a link pointing to it. Since you don’t link to your variation pages from anywhere from the site or else-where, search engines won’t know that those pages even exist. For the truly paranoid, you can instruct search engines not to index you variations pages. You can do it either using no-index meta attribute or by using robots.txt
Lesson: A/B split testing has absolutely NO negative impact on your SEO efforts!
Rest assured, you can use Visual Website Optimizer without worrying about its impact on search engine rankings. If you have a specific SEO and A/B testing related question, let me know in comments below.
UsabilityWeb.nl is a Dutch blog/magazine on the topics of usability, user experience and human-centered design. It’s an initiative by Concept7, which is one of our agency customers. During last few months, they had been running couple of split tests on the magazine site using Visual Website Optimizer. As you will read, they increased the conversion rate by as much as 80% simply by testing a different phrase. This result is similar to our previous A/B testing case studies where a small change indeed resulted in a significant improvement in conversions. We are proud to share a joint-case study on UsabilityWeb.nl A/B testing below:
You can also download the A/B testing case study in PDF format. Please feel free to share, tweet, embed the presentation to spread the word about potential of A/B testing for improving conversions.
If you want to read the transcript for the presentation, here it is:
