I love split testing - Visual Website Optimizer Blog » split testing

Do human photos on a landing page increase sales and conversions?

Paras Chopra — Wed, 18 May 2011 10:13:22 +0000

Starting with a new series on this blog called Revisiting the Classics, I will re-look at our A/B testing case studies and throw some new light on it (with additional commentary). The case study that I picked for this post relates to human photos on landing pages and whether they increase conversion rate. Before reading rest of the post, make a guess: do human photos (male/female) online really have any subconscious effect on visitors?

What research says about human photos on websites?

There has been significant scientific research on this topic particularly for eCommerce websites. When you are considering a purchase decision, the main issue online is of trust. How do you know if a particular website is trust worthy? Taking lessons from TV advertising and general marketing principles, people running eCommerce websites think that associating photos of people with products engenders trust. Of course, human photos = increased trust is just a hypothesis. But is it really true?

To answer this, I analyzed several research papers (which are published in peer-reviewed scientific journals). Following are some of the main ones (along with excerpts):

The Human Face of E-Business: Engendering Consumer Initial Trust Through the Use of Images of Sales Personnel on E-Commerce Web Sites: initial trust was found to be enhanced for Web sites employing photographs and video clips compared to control Web sites lacking such images
Shiny happy people building trust?: photos on e-commerce websites and consumer trust: trustworthiness of low-trust sites can be boosted by adding a photo
Exploring Human Images in Website Design Across Cultures: A Multi-Method: results indicate that websites with images that include facial characteristics are more positively received by users than images with no facial features or no human images at all
Trust at first sight? A test of users’ ability to identify trustworthy e-commerce sites: a photo only had an effect on participants’ first impression of a vendor

Not just scientific research, real world A/B tests that I have seen also corroborate the hypothesis of human photos may increase conversion rate. Two of our customers used our A/B testing tool (Visual Website Optimizer) to test presence of human photos and its impact on conversions. Following sections detail what they did and what they found out.

Photos v/s Paintings (A/B test #1)

Medalia Art sells Brazilian and Caribbean art online. Since they are an online art shop, they showcase paintings from famous artists on their homepage. They decided to test what happens if paintings are replaced by the artists’ photos.

The goal for this A/B test was to increase visitor engagement (defined as a click on any link on the homepage; it is the inverse of the bounce rate). They didn’t use sales as a conversion rate as many of their sales happen after interaction on the telephone with the customer.

Here are the screenshots of the control (paintings) and the winning version (photos).

- Original (with paintings)-

- Variation (with photos)-

Results?: Medalia found out that with paintings the conversion rate was 8.8% but if paintings are replaced with artist photos, the conversion rate increases to whopping 17.2%. That’s an increase in conversion rate of more than 95%! (results were statistically significant) How cool is that for doing a slight change?

Photo v/s Generic icon (A/B test #2)

Another user, Jason Thompson, conducted an A/B test on his blog to see if replacing contact icon with his own photo would lead to more people contacting him. Following is screenshot of original and variation:

Original (with icon)	Variation (with photo)

3.7% conversion rate	5.5% conversion rate

Results clearly show that the version with Jason’s photo had 48% more conversions as compared to generic icon. (Again, results are statistically significant).

Jason commented on the test results:

People want to connect with other people emotionally, the photo makes that emotional connection so much easier and as the test is proving, drives people to the contact form more than a nondescript icon.

So, should you use human photo on your website?

There are hundreds of more research studies and probably more A/B test results that you can read but from what I have read and observed, here are some salient points about human photos on websites:

Human photos on a website definitely have positive impact on visitor’s first impression of trustworthiness
Human photos with focus on face have much better impact (as emotional connect is stronger)
Photos should be “real”. Visitors can tell when you are using stock photos
Most important conclusion: human photos are not panacea for all websites. Best is to A/B test photos v/s no photos on your website. Many research papers proved that in some cases, human photos may actually have a negative impact!

Hope you found this article useful. If you have any comments or suggestions, please let me know.

A/B testing is not snake oil

Paras Chopra — Fri, 25 Mar 2011 16:00:14 +0000

Recently on Hacker News, someone commented that A/B testing has become like snake oil, making grand promises to be a panacea for increasing conversion rates. This comment troubled me not because the commenter was wrong (he was!) but because how certain people view A/B testing. Let me put this straight and clear:

A/B testing does NOT guarantee increase in sales and conversions

In fact, we recently featured a guest post by Noah of Appsumo who revealed that only 1 out of 8 split tests that they run produce any actionable results. Yep, read that once again: only 1 out of 8 split tests work. Snake oil guarantees to be a cure for all diseases; A/B testing has no such guarantees.

I think the reason why some people would view split testing suspiciously is because we release a lot of A/B testing case studies. In fact, with a user base so large (6000+ at the time of writing), we see many of our customers getting good results all the time and they if want to share those results with the world, we publish them on our blog.

So, when one week we publish a case study titled 3 dead-simple changes increased sales by 15% and the next week we publish another case study titled How WriteWork.com increased sales by 50%, it naturally makes some people assume using Visual Website Optimizer to do A/B testing would ensure a growth in sales. No, using Visual Website Optimizer won’t guarantee anything. It is just an A/B testing tool (albeit, a very good one!) but the real work done is always by the craftsman, not the tool s/he uses.

Survivorship Bias: it helps to know a thing or two about it

Most success stories or case studies (be it about A/B testing, getting rich or losing weight) that you read about on the Internet suffer from something known in psychology as survivorship bias. In context of A/B testing, what it means is simple:

If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.

Obviously, 99 case studies about how A/B testing did not produce any results in unexciting and will bore our blog readers. Hence, this post is a short reminder about all those case studies that don’t produce results.

Why publish A/B testing case studies at all

The reason we publish A/B testing case studies is to show potential of what can be achieved using A/B testing. It is similar to holding an art exhibition for Picasso paintings which inspires people to appreciate the work and perhaps pick up brush themselves to start painting hoping to achieve level of success that Picasso achieved. To reiterate, just as picking up a brush and canvas does not guarantee good art, doing A/B testing does not guarantee increase in sales.

In fact, specifically to A/B tests, replicating them almost never works. So, if you read a case study of how video increases conversions by 46% and then implement it on your site hoping for the same magical increase, it may or may not happen. What worked for them may not work for you! So, why publish case studies at all? Here’s why:

A/B testing case studies give ideas on what you can test on your website

Yep, that’s all there is to A/B testing case studies.

And, yes, I hope I convinced you that A/B testing is not snake oil!

Revenue Tracking for A/B testing: exciting new feature in Visual Website Optimizer

Paras Chopra — Wed, 23 Feb 2011 15:53:53 +0000

We are extremely proud to release a brand new feature in Visual Website Optimizer: Revenue Tracking. This is a significant new development for our product because it means now in addition to tracking conversion rate (for multiple goals such as clicks on links, visit to pages, form submissions, engagement, etc.), you can track various revenue metrics as well (including revenue per visitor, total revenue, average order value, etc.)

Why you should track revenue in your split tests?

Revenue tracking for A/B split tests is utmost important when a monetary transaction is involved (either immediately or at the end of conversion funnel) and a conversion goal may have different monetary values. In some cases, higher conversion rate may translate into lower revenue, so basing all decisions merely on conversion rate can lead you to choosing a variation that actually decreases your revenue.

Let’s imagine that a software / eCommerce company is running a split test on pricing page where (in a variation) lowest plan of $2o has been removed. For example, imagine running a test such as below:

If you just measure conversion rate, you can expect to see following results (lower conversion rate for variation since we removed smallest plan in it):

Control (original page): 1.2%
Variation (smallest plan removed): 0.8%

However, if you start measuring revenue for this split test, you can see that variation actually resulted in higher revenue per visitor:

Control (original page): 28 cents per visitor revenue and $1400 total revenue
Variation (smallest plan removed): 36 cents per visitor revenue and $1800 total revenue

Naturally, you would want to choose a variation which results in higher per visitor revenue even if total number of conversions and conversion rate decreases. That’s why revenue tracking is so important for A/B, split and multivariate tests. It lets you measure impact of your test variations on a metric that drives your business: revenue.

How to do Revenue Tracking in Visual Website Optimizer

We believe revenue tracking shouldn’t be hard and that’s why we have tried to make it dead-simple to implement it in Visual Website Optimizer. All you need to do is this:

Create A/B, split or multivariate test
On goals page, select ‘Revenue Tracking’ as goal and enter the URL of the goal as the page where you will have access to visitor’s revenue number (it is typically available on thank you page after a successful payment / transaction)
On that thank you page, you simply need to copy-paste default VWO code and output a JavaScript variable _vis_opt_revenue with value equal to total revenue generated by the visitor. For example, if the visitor purchased 2 items (one $15 and the other $7), simply output $22 in the variable. If visitor only bought one item (say $10), output $10 only. That’s it. You don’t have to be concerned about which variation visitor saw or what individual items did the visitor bought. You just need to output total revenue and Visual Website Optimizer will take care of the rest.

For example, if the page where we redirect after successful plan purchase is http://visualwebsiteoptimizer.com/thank-you.php then all we need to do is to copy-paste default VWO code there and also output following code:

Of course, the value of revenue (49 in this case) will change dynamically depending on actual revenue generated by the visitor. If you already have revenue tracking setup with your web analytics tool (such as Google Analytics), extending it for Visual Website Optimizer should be a piece of cake. If not, just ask your IT guy, he would know where and how to output revenue for tracking. All in all, implementing revenue tracking in Visual Website Optimizer is quite simple.

How do Revenue Tracking reports look like?

Once revenue tracking is in place, you will see extensive reports like following:

As you can see, reports show you various metrics for different variations:

Revenue per visitor (the basic metric that determines performance of variation)
Total revenue
Total number of conversions (you can calculate average order value as Total Revenue / Total conversions)

Hope you like this new feature! Let us know if you need any help or assistance with revenue tracking in Visual Website Optimizer.

How to increase newsletter signups by 28%? Ask users to ‘watch a video’ or ‘get instant access’

Paras Chopra — Thu, 06 Jan 2011 11:27:58 +0000

Last week we ran an A/B test contest where you had to guess which button increased newsletter signups by 28%. Was it ‘watch a video’ or was it ‘get instant access‘? We got a total of 51 submissions (tweets + blog comments) and would you believe it: 51% said ‘Get Instant Access‘ would win while 49% said ‘Watch a video‘ will win. This is incredible and clearly demonstrates how hard is it to guess what is going to work on your website!

Before announcing the correct answer, we’d like to announce the winner of $3000 worth Visual Website Optimizer subscription. Please join me for the drum roll… and the winner of this contest is: Stephan Schubert and here is the winning tweet. Congratulations, Stephan!

‘Watch a video’ v/s ‘Get Instant Access’ – 28% increase in signups

I recently interviewed The Social Man (one of the Visual Website Optimizer customers) on their successful split test. As you will read, this test is typical example of how a single change (in call-to-action) can result in significant increase in conversions. Almost all our case studies make the same point about A/B testing which is worth repeating twice: (seemingly) small changes can increase conversions on a landing page. So, you should always be testing.

Here is the full interview:

1. What was the conversion goal of the split test?

To convert visitors to email newsletter subscribers, and to get them to see a video about how a guy can learn to “talk like a ten”.

2. On which page did you run the test?

http://www.becomeunbreakable.com/1/talk-like-a-ten/indexVWO.php

(Please note – this is a different page than the original, but keeps all the same content. We track lead sources and use different landing pages for traffic from different sources)

3. What is the traffic source? Organic, direct, PPC, etc.?

Cold traffic from a dating website – CPM. We tested a lot of different bids and were very surprised about the most profitable bid price.

4. Which part of page did you select for the test and what variations did you test?

We initially ran a test with three different variations each of headline and sub-head. Once we found the best combination for those, we split the “Submit” button on a whim. One said “Free Instant Access” and one said “Watch The Video”. See images below:

Version A: Get Instant Access (11.9% conversion rate)

Version B: Watch the Video (15.3% conversion rate)

Editor’s note: so now you know that ‘Watch the video‘ increased conversions from 11.9% to 15.3%. A total increase of 28%!

5. Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?

Pushing for the action of “Watch The Video”. The subhead that won (“Get The Free ‘Talk Like A Ten’ Video Now!!!”) also mentioned that they could “get” the “video”.

6. What results did you get? Were you surprised by the results?

15% opt-ins for cold traffic off this particular site is amazing – We’re running ads/offers into a few very nichey demographics on that site that generate huge CTR’s (.300-.400; in comparison, the CTR on this campaign is .086 for our best creative) and haven’t beaten this conversion for any other squeeze page.

7. Any lessons which can be derived from your test?

I’m reminded of the parable where an old friend of Henry Ford’s said “Henry, why don’t you ever buy any bolts from me?” to which Mr. Ford replied “Heck Joe, you’ve never asked!” It’s always worth including action verbs like “get”, “watch”, “download”, “enter” etc. in tests.

8. How valuable was Visual Website Optimizer for this test?

It’s so much easier – and more powerful – than Google’s website optimizer. We’re finding it to be an indispensable tool that made me excited about running tests again.

9. Short background information about your business.

The Social Man is in business is to make men awesome. We have a variety of products and coaching packages that cover everything from flirting over text messaging, to how to be sexier, to overcoming social anxiety. I previously ran an enterprise software company, and feel very blessed to be doing something I love that’s employs passionate people and has helped so many men. My business, Turnseven, Inc., manage a few other niche brands in addition to The Social Man.

Best A/B Testing Case Studies of 2010

Paras Chopra — Mon, 20 Dec 2010 12:58:17 +0000

Boy, what a year 2010 has been for A/B testing industry and Visual Website Optimizer. We saw tens and hundreds of small/medium sized businesses increase sales and conversions on their websites. Our repository of A/B testing case studies is the proof that A/B testing works! To celebrate the excitement of 2011, I have complied the list of our best A/B testing case studies in 2010 (in order of bestness). Hope they will inspire you to kick-start A/B testing campaign in your organization in 2011.

Removing a signup form actually increased signups by 60%

Original Landing Page (with embedded signup form)

Variation (no signup form): 60% increase in signups!

Who would have thought that removing a signup form from signup page can actually increase conversions by 60%? Vendio, an Alibaba.com company, used Visual Website Optimizer to do an A/B test on their signup landing page. In one version, they removed the signup form altogether and instead wrote about benefits of signing up. The end result was that the version with no signup form eventually saw 60% more signups. Click here to read the full case study (includes screenshots)

Two magical words improve conversion rate by 28%

Control: 14.5% conversion rate

Variation: 28% improvement conversion rate!

Soocial tested multiple variations of their sign up button in order to increase clickthroughs to the registration form. Variations included call-to-actions such as “Sign up for free”, “Free sign up”, etc. The A/B test results revealed that one variation increased conversions dramatically. Click here to read the full case study (includes screenshots)

Human photos double conversion rate

Control (paintings)

Variation (photos): 95% increase in CTR!

In two separate A/B tests, it was found out that human face photos doubled the conversion rate on a website. The result was striking because the tests were conducted by two completely unrelated companies, yet they arrived on the same conclusion. Click here to read the full case study (includes screenshots)

Downloads increased by 85% by testing tour page

Original (control)

Redesigned (variation): 85% increase in downloads!

You Need A Budget makes easy-to-use personal financial management software. They split tested their product tour page by including more screenshots, adding a prominent testimonial and reducing text clutter. The redesigned page increased downloads of the free trial by 85%. Click here to read the full case study (includes screenshots)

Redesign of sales page improves sales by 20%

Original version

Redesigned version: 20% increase in sales!

AquaSoft redesigned their sales page by giving it a modern, clean look. They also added trust building elements such as money-back guarantee, assurance seal, etc. Using Visual Website Optimizer, they effortlessly tested old v/s new sales page. After multiple phases of testing, they successfully increased total sales by 20%. Click here to read the full case study (includes screenshots)

What’s in store for 2011?

Well, we already have a lineup of many interesting split testing case studies. From product pages to eCommerce sites. From effect of video on signups to effect of number of steps in checkout process on sales. We have a case study lined up for all these situations.

To keep updated with our latest A/B testing case studies, subscribe blog updates through email (powered by Feedburner).

Killer idea: play the split testing game with your boss and colleagues

Paras Chopra — Thu, 16 Dec 2010 11:17:10 +0000

Let’s admit it: your boss has certain opinions on the website design, the usability consultant has some “best-practices” suggestions and the sales team wants specific changes to increase leads. Heck, even your office receptionist thinks that you could add Facebook and Twitter buttons on the homepage. Needless to say, everyone in your office has an opinion on website design and if you are the one with Photoshop/HTML skills, it can drive you crazy fairly quickly. So, how to cope up with suggestion/feedback overload all the while trying to increase conversion rate on the website?

Play the Split Testing Game

Rules of the game are pretty simple. You declare a split testing tournament at the Friday office lunch to win a booty of say $100 (or any other motivating prize). Anyone in your office (or team) with an opinion on site design gets to create a variation for a split test. You then pit all variations against each other in a live split test and quite obviously the person who came up with winning variation wins the prize booty!

Why is this idea killer?

This game embraces the multi-dimensionality of the team because everyone (from sales team to your accountant) gets to contribute one variation. Split testing is always full of surprises so it is hard to guess what will work. So, what’s better than trying out different ideas from your colleagues.
You get a real chance to defeat your boss Your boss thinks he is smarter than you are? Challenge him/her in a split testing tournament and prove conclusively who deserves to be the real boss!
It’s fun and boosts team spirit. Nothing is more fun than tracking the leader-board of whose variation is converting the best.

This strategy brings out the best talent from across the team and helps you test diverse ideas at once. Since inputs for improving conversion rate and sales comes from different team members, this also helps you avoid designer’s block (of coming up with highly converting ideas and designs).

How to use Visual Website Optimizer for playing Split Testing Game

Visual Website Optimizer is the perfect tool for playing this game. Here’s why:

You can easily create separate logins for each of the team-mates taking part in the game.
Best feature: using WYSIWYG editor, your boss himself can create variations for split testing. This means no extra burden on you for organizing the split testing tournament!
Reports are realtime which means you can put the reports (with auto refresh) on the cafeteria LCD to keep track of who is winning!
Based on statistical significance, you automatically get email notifications as soon as a winning variation is found.

So, are you game for the split testing game?

Do you think this is a killer idea? Have you ever tried this strategy in your office? At Wingify, we keep playing this game (after all, we make split testing tool). You also try this and let us know your thoughts and opinions. We have heard that the split testing game works like wonder to increase conversion rate and sales.

Get ideas for A/B testing from website evaluation by experts

Paras Chopra — Tue, 16 Nov 2010 09:24:00 +0000

Let’s admit it: getting ideas for A/B and multivariate tests is a hard job! It becomes even harder because of your familiarity with your website or landing page. Since you breathe-and-eat your website every day and know all its intricacies, it becomes incredibly hard to think of improvements. This phenomenon even has a medical name: Blind Spot.

To get ideas on what you can test in your next A/B or multivariate test, we have a A/B testing case study search engine called Ideafox. Also, earlier on this blog I talked about a service called Feedback Army which lets you have feedback from 10 people for $10 (I see the price is $15 now). Feedback Army is great service and we had received lots of relevant feedback from that site (in addition to some actionable ideas for A/B testing). As also noted in original post, the only drawback of this service is that people who give feedback are hired from Mechanical Turk, so feedback length is short and doesn’t go into too much depth (since those people are only paid 50 cents per feedback). Inspite of this, Feedback Army has a great ROI for an investment of $15.

What if you need feedback on your website from a professional?

Enter Concept Feedback, the website to get expert website evaluations and feedback. It is a community of design, usability and conversion lovers who comment and give feedback on your website and landing page. They have experts for three categories of feedback:

Design
Usability
Startegy

Experts have years of experience in their respective fields along with professional degrees in design. So, you can be guaranteed to get top-notch feedback on your website from Concept Feedback. Of course, all this comes at a price. You need to shell out $100 per expert feedback. So if you want feedback from 5 experts, that is going to cost you $500. It’s a little expensive but as you will read below, the quality and amount of feedback that you get from an expert has way more value than $100.

Evaluation of Visual Website Optimizer website on Concept Feedback

To try out the service, I evaluated Concept Feedback to get feedback on Visual Website Optimizer website design. Oh boy, I was pleasantly surprised with the quality of feedback on as many five different pages of our site: homepage, signup page, features page, case study page and blog. Here is some of the feedback produced verbatim:

“I would do something to visually separate this section of the site from the rest. Perhaps make the background blue and reverse the text in white, make the call to action orange?”
“I would reduce the # of statistics in the call to action area. While these are great to display and to the point, 4 might be a bit much. Consider showing 1-3. Perhaps in a slider form.”
“I would make your headlines sans-serif to help readers visually distinguish and scan your content easily. This goes for h1’s, h2’s, h3’s etc. Making them a different color (besides black) will also help.”

All these comments come with a screenshot of your page so you can see where exactly the expert is referring to when he says “make your headlines sans-serif” (in this case, he was talking about our blog).

What’s more? The expert even quickly created a mockup of what according to him our site should look like. Here is the version he redesigned:

I think this redesign is definitely an improvement over existing design. What do you think?

Conclusion

For getting ideas for A/B and multivariate tests, nothing works better than feedback and evaluation services such as Feedback Army and Concept Feedback. The ‘strategy’ feedback type in Concept Feedback is especially relevant to A/B tests because there you have access to experts who have experience and knowledge in conversion rate optimization.

So, next time you are stuck and can’t think of anything to improve on your site, you know where to get feedback from.

New Feature: Custom URLs for your A/B and Multivariate tests

Paras Chopra — Tue, 09 Nov 2010 15:09:57 +0000

Visual Website Optimizer has a feature that allows running of same A/B or multivariate test on multiple pages at once. For example, if your site has several product categories and you would like to run a test only on one category, all you need to do is to enter a URL pattern like http://example.com/shoes/* in Visual Website Optimizer.

The URL pattern above will make the test run on all pages in the shoes directory. This feature is very handy because you can test common call-to-actions, headers, footers, sidebars, etc. consistently on multiple pages at one go. What’s more, a consistent variation is shown on all the pages in a test. So if a user sees red variation on one shoes page, he will always see the same red variation on all other shoes pages.

Sometimes (primarily due to SEO reasons) there is no clear pattern in the URLs. That is, it may be possible that there are several independent URLs for category or product pages without any apparent pattern. For example, if there are URLs such as http://example.com/adidas/, http://example.com/reebok/, http://example.com/microsoft/, http://example.com/adobe/, etc. and you wish to run the test on only on shoes pages (the first two URLs), how are you going to do that? Until now it wasn’t possible to setup such a test, but with our new feature Custom URLs it is very simple to setup such tests.

How do I setup a custom URL test?

Actually, there are three easy steps if you want to use custom URLs:

Come up with an imaginary custom URL for all pages that need to be tested: note that the actual custom URL may not exist as we just want to come up with an identifier. For example, custom URL can be http://example.com/my_shoes_pages
Use that URL in VWO: while creating a test in VWO, enter that custom URL in the field where it asks whether test is to be run on multiple pages (URL pattern field)
Output that custom URL on your page in _vis_opt_url JavaScript variable: since the custom URL is imaginary, you need to output it in a _vis_opt_url variable on all pages you need that page to run the test.You also need to modify (common) VWO code snippet a little to accommodate custom URLs.

Example Code:

On all pages where you need to run a specific test, output _vis_opt_url variable and modify VWO code as following (bold parts are extra modifications):


...
// Only on pages where this test is to be run
var _vis_opt_url = 'http://example.com/my_shoes_pages';
...
...
...
// Common VWO Code
var _vis_opt_url = typeof(_vis_opt_url)=="undefined" ? document.URL : _vis_opt_url;
....
'&url='+encodeURIComponent(_vis_opt_url)
+'&random='+Math.random()
...

Sounds Complicated? We’ll help you implement

We do realize that the implementation is bit complex but we wanted to roll out the feature to all customers (many enterprise customers have been using it already). We are here to help you implement VWO code for custom URLs. Simply contact support@wingify.com if you need our help and assistance.

Related posts:
Visitor Segmentation for A/B and Multivariate Tests

Bunch of small, new features in Visual Website Optimizer

Revenue Tracking for A/B testing: exciting new feature in Visual Website Optimizer

A/B testing with competing goals: newsletter CTR increased by 190%, but clicks on RSS feed…

Paras Chopra — Tue, 19 Oct 2010 12:56:45 +0000

Many times A/B testing is not limited to a single conversion goal. In fact, your test variations usually affect many different conversion goals on your site such as free trial signups, paid signups, newsletter subscription, etc. Measuring all these goals for a test is important because a variation may work brilliantly for one goal (e.g. it increases free signups) but may perform worse for your other goals (e.g. decreases your paid signups). If you don’t measure multiple goals for your A/B test, you are essentially flying blind in the dark and may end up making wrong decisions (based on a single conversion goal).

Case Study

A brilliant example of this is a recent A/B test by one of our customers: Guido Jansen (a Magento specialist). He runs Dutchento.org which is the official Dutch community for Magento CMS. The goal of his recent A/B test was to increase subscriptions for Dutchento’s newsletter and RSS subscription. The call-to-action for subscription is located in a box on all pages of the site. See the control version of subscription box below and note that there is no explicit incentive for a visitor to subscribe:

Guido tested a variation of this subscription box which included a title and some benefits in a bullet list. Here is his hypothesis for increasing subscriptions:

People don’t just subscribe to a newsletter or newsfeed for nothing, you should convince them it has added value above just visiting the website. So what I wanted to test is if adding convincing reasons to subscribe would increase the newsletter and newsfeed subscription rate. I measured the impact of the convincing reasons on clicks on both the newsletter and newsfeed links.

Here is how his variation (of subscription box) looks like:

Results

As expected, Guido saw a significant improvement of the newsletter click rate (+190.31%). However, newsfeed click rate decreased (-44.46%) which did surprise him (and us!). He expected that the convincing reasons would affect both positively, but apparently it had a negative effect on newsfeed clicks.

The reason why clicks on newsfeed decreased is not clear but we believe that the benefits in variation were so compelling that visitors chose to get the blog updates via email (where they will be sure to read them) rather than RSS reader (where they may miss them). A great way to get more insight into this would be to randomize the position of newsfeed / newsbrief to eliminate the positional effect of those links.

Testimonial

Guido used Visual Website Optimizer for A/B testing and here is what he had to say:

Visual Website Optimizer was very valuable [for testing]. It’s the easiest A/B and Multivariate testing tool I know. It’s great not to be dependent on the development department to create and run your tests.

Key Lesson

Whatever be the actual reason of decreased clicks on RSS feed, one key lesson is jumps out of the case study: always measure multiple conversion goals in your test. Relying on a single conversion goal hides a lot of valuable information from you. So, make sure you add multiple goals to your next A/B test.

Do you have more examples of using multiple goals in A/B test? What are your thoughts on this case study? Please let us know by leaving a comment below – will be happy to discuss!

Impact of A/B testing on Search Engine Optimization (SEO)

Paras Chopra — Thu, 07 Oct 2010 13:30:12 +0000

A lot of our potential customers are concerned whether A/B or multivariate testing will have an effect on their search engine rankings. They (obviously) aren’t looking to improve their rankings by using A/B testing, rather their concerns relate to the myth that testing may negatively impact search engine optimization efforts.

Broadly, there mainly two major concerns:

Content Cloaking: the act of showing different content to search engine bots and actual human visitors. A/B testing software (such as Visual Website Optimizer) swap content on the page using JavaScript, so some users see different content (of variations) and bots see original page only.
Duplicate Content: the act of copying content from elsewhere and hosting it on your site. Search engines penalize for such stealing of content because you cannot expect to rank on keywords for the content that isn’t yours. Again, many A/B and multivariate testing software (including VWO) have an option where you can redirect site traffic to different variations of a page which raises this concern.

Why A/B testing is not content cloaking

Cloaking content was all rage back in the early days of search engines (think 1990s). In those days, SEO was all about keyword stuffing. So, smart SEO geeks used to display a page full of keywords when a search engine bot visited to crawl / index the page. While if a user visited the same page, they displayed the default (normal) version. This strategy of keyword stuffing used to work like wonders so naturally search engines started devising clever ways to detect and penalize such cloaking and it may appear that pages where A/B testing is on will also get penalized because it is kind of cloaking.

Good news is that thanks to Google’s PageRank algorithm, keyword stuffing no longer works. So, there is little incentive for search engines to penalize content cloaking. Moreover, unlike yesteryears’ static HTML sites, today’s search engines have come to expect highly dynamic AJAX driven sites. So, they no longer consider swapping content dynamically as cloaking. If you use a multivariate testing software to swap different parts of your page, it is not cloaking! Also, all the major search engines (Google, Yahoo, Bing, etc.) are well versed with concept of A/B split testing and they do many such tests on their own sites every single day. You wouldn’t expect search engines to penalize others for doing the same activity that they themselves do regularly, would you?

Key lesson: Doing A/B testing by dynamically swapping elements on page (using JavaScript) is NOT content cloaking. It is absolutely safe and should not have any negative impact on Search Engine Optimization efforts.

Split URL testing (redirecting traffic to multiple versions) and the issue of content duplication

As I mentioned above, this is another issue that worries many prospective customers. While one approach for split testing is to dynamically swap page elements (such as headlines, images, text, etc.), another approach is to host different variations of page on your website. So, if you want to split test product.html, you will create variation pages and host them as product1.html, product2.html, etc. A split testing software will redirect all traffic coming to product.html to the variation pages. Since product1.html and product2.html will have similar content as product.html, it worries some people that search engines will consider them duplicate content and penalize rankings.

An important point to note here is that search engines only penalize if you steal or host content from a different domain; here all variations pages host on your own site and you OWN that content and are free to do with it whatever you want to. Many dynamic websites (shopping carts, directories, etc.) today host the same content in different formats. Search engines don’t penalize them because that unique content is only found on their domain and no where else. Same is the case with your split testing URLs. You don’t get negatively impacted by it because you haven’t stolen that content — it is all yours!

A more important point is answer to this question: how will search engines come to know that there are different variations of your main page? Your variation pages can only be reached if there is a link pointing to it. Since you don’t link to your variation pages from anywhere from the site or else-where, search engines won’t know that those pages even exist. For the truly paranoid, you can instruct search engines not to index you variations pages. You can do it either using no-index meta attribute or by using robots.txt

Lesson: A/B split testing has absolutely NO negative impact on your SEO efforts!

Rest assured, you can use Visual Website Optimizer without worrying about its impact on search engine rankings. If you have a specific SEO and A/B testing related question, let me know in comments below.