Starting with a new series on this blog called Revisiting the Classics, I will re-look at our A/B testing case studies and throw some new light on it (with additional commentary). The case study that I picked for this post relates to human photos on landing pages and whether they increase conversion rate. Before reading rest of the post, make a guess: do human photos (male/female) online really have any subconscious effect on visitors?
There has been significant scientific research on this topic particularly for eCommerce websites. When you are considering a purchase decision, the main issue online is of trust. How do you know if a particular website is trust worthy? Taking lessons from TV advertising and general marketing principles, people running eCommerce websites think that associating photos of people with products engenders trust. Of course, human photos = increased trust is just a hypothesis. But is it really true?
To answer this, I analyzed several research papers (which are published in peer-reviewed scientific journals). Following are some of the main ones (along with excerpts):
Not just scientific research, real world A/B tests that I have seen also corroborate the hypothesis of human photos may increase conversion rate. Two of our customers used our A/B testing tool (Visual Website Optimizer) to test presence of human photos and its impact on conversions. Following sections detail what they did and what they found out.
Medalia Art sells Brazilian and Caribbean art online. Since they are an online art shop, they showcase paintings from famous artists on their homepage. They decided to test what happens if paintings are replaced by the artists’ photos.
The goal for this A/B test was to increase visitor engagement (defined as a click on any link on the homepage; it is the inverse of the bounce rate). They didn’t use sales as a conversion rate as many of their sales happen after interaction on the telephone with the customer.
Here are the screenshots of the control (paintings) and the winning version (photos).
- Original (with paintings)-
- Variation (with photos)-
Results?: Medalia found out that with paintings the conversion rate was 8.8% but if paintings are replaced with artist photos, the conversion rate increases to whopping 17.2%. That’s an increase in conversion rate of more than 95%! (results were statistically significant) How cool is that for doing a slight change?
Another user, Jason Thompson, conducted an A/B test on his blog to see if replacing contact icon with his own photo would lead to more people contacting him. Following is screenshot of original and variation:
|
Original (with icon) |
Variation (with photo) |
 |
 |
|
3.7% conversion rate |
5.5% conversion rate |
Results clearly show that the version with Jason’s photo had 48% more conversions as compared to generic icon. (Again, results are statistically significant).
Jason commented on the test results:
People want to connect with other people emotionally, the photo makes that emotional connection so much easier and as the test is proving, drives people to the contact form more than a nondescript icon.
There are hundreds of more research studies and probably more A/B test results that you can read but from what I have read and observed, here are some salient points about human photos on websites:
Hope you found this article useful. If you have any comments or suggestions, please let me know.
Recently on Hacker News, someone commented that A/B testing has become like snake oil, making grand promises to be a panacea for increasing conversion rates. This comment troubled me not because the commenter was wrong (he was!) but because how certain people view A/B testing. Let me put this straight and clear:
In fact, we recently featured a guest post by Noah of Appsumo who revealed that only 1 out of 8 split tests that they run produce any actionable results. Yep, read that once again: only 1 out of 8 split tests work. Snake oil guarantees to be a cure for all diseases; A/B testing has no such guarantees.
I think the reason why some people would view split testing suspiciously is because we release a lot of A/B testing case studies. In fact, with a user base so large (6000+ at the time of writing), we see many of our customers getting good results all the time and they if want to share those results with the world, we publish them on our blog.
So, when one week we publish a case study titled 3 dead-simple changes increased sales by 15% and the next week we publish another case study titled How WriteWork.com increased sales by 50%, it naturally makes some people assume using Visual Website Optimizer to do A/B testing would ensure a growth in sales. No, using Visual Website Optimizer won’t guarantee anything. It is just an A/B testing tool (albeit, a very good one!) but the real work done is always by the craftsman, not the tool s/he uses.
Most success stories or case studies (be it about A/B testing, getting rich or losing weight) that you read about on the Internet suffer from something known in psychology as survivorship bias. In context of A/B testing, what it means is simple:
If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.
Obviously, 99 case studies about how A/B testing did not produce any results in unexciting and will bore our blog readers. Hence, this post is a short reminder about all those case studies that don’t produce results.
The reason we publish A/B testing case studies is to show potential of what can be achieved using A/B testing. It is similar to holding an art exhibition for Picasso paintings which inspires people to appreciate the work and perhaps pick up brush themselves to start painting hoping to achieve level of success that Picasso achieved. To reiterate, just as picking up a brush and canvas does not guarantee good art, doing A/B testing does not guarantee increase in sales.
In fact, specifically to A/B tests, replicating them almost never works. So, if you read a case study of how video increases conversions by 46% and then implement it on your site hoping for the same magical increase, it may or may not happen. What worked for them may not work for you! So, why publish case studies at all? Here’s why:
A/B testing case studies give ideas on what you can test on your website
Yep, that’s all there is to A/B testing case studies.
And, yes, I hope I convinced you that A/B testing is not snake oil!
We are extremely proud to release a brand new feature in Visual Website Optimizer: Revenue Tracking. This is a significant new development for our product because it means now in addition to tracking conversion rate (for multiple goals such as clicks on links, visit to pages, form submissions, engagement, etc.), you can track various revenue metrics as well (including revenue per visitor, total revenue, average order value, etc.)
Revenue tracking for A/B split tests is utmost important when a monetary transaction is involved (either immediately or at the end of conversion funnel) and a conversion goal may have different monetary values. In some cases, higher conversion rate may translate into lower revenue, so basing all decisions merely on conversion rate can lead you to choosing a variation that actually decreases your revenue.
Let’s imagine that a software / eCommerce company is running a split test on pricing page where (in a variation) lowest plan of $2o has been removed. For example, imagine running a test such as below:
If you just measure conversion rate, you can expect to see following results (lower conversion rate for variation since we removed smallest plan in it):
However, if you start measuring revenue for this split test, you can see that variation actually resulted in higher revenue per visitor:
Naturally, you would want to choose a variation which results in higher per visitor revenue even if total number of conversions and conversion rate decreases. That’s why revenue tracking is so important for A/B, split and multivariate tests. It lets you measure impact of your test variations on a metric that drives your business: revenue.
We believe revenue tracking shouldn’t be hard and that’s why we have tried to make it dead-simple to implement it in Visual Website Optimizer. All you need to do is this:
For example, if the page where we redirect after successful plan purchase is http://visualwebsiteoptimizer.com/thank-you.php then all we need to do is to copy-paste default VWO code there and also output following code:
<script>
_vis_opt_revenue=49;
</script>
Of course, the value of revenue (49 in this case) will change dynamically depending on actual revenue generated by the visitor. If you already have revenue tracking setup with your web analytics tool (such as Google Analytics), extending it for Visual Website Optimizer should be a piece of cake. If not, just ask your IT guy, he would know where and how to output revenue for tracking. All in all, implementing revenue tracking in Visual Website Optimizer is quite simple.
Once revenue tracking is in place, you will see extensive reports like following:
As you can see, reports show you various metrics for different variations:
Hope you like this new feature! Let us know if you need any help or assistance with revenue tracking in Visual Website Optimizer.
Last week we ran an A/B test contest where you had to guess which button increased newsletter signups by 28%. Was it ‘watch a video’ or was it ‘get instant access‘? We got a total of 51 submissions (tweets + blog comments) and would you believe it: 51% said ‘Get Instant Access‘ would win while 49% said ‘Watch a video‘ will win. This is incredible and clearly demonstrates how hard is it to guess what is going to work on your website!
Before announcing the correct answer, we’d like to announce the winner of $3000 worth Visual Website Optimizer subscription. Please join me for the drum roll… and the winner of this contest is: Stephan Schubert and here is the winning tweet. Congratulations, Stephan!
I recently interviewed The Social Man (one of the Visual Website Optimizer customers) on their successful split test. As you will read, this test is typical example of how a single change (in call-to-action) can result in significant increase in conversions. Almost all our case studies make the same point about A/B testing which is worth repeating twice: (seemingly) small changes can increase conversions on a landing page. So, you should always be testing.
Here is the full interview:
1. What was the conversion goal of the split test?
To convert visitors to email newsletter subscribers, and to get them to see a video about how a guy can learn to “talk like a ten”.
2. On which page did you run the test?
http://www.becomeunbreakable.com/1/talk-like-a-ten/indexVWO.php
(Please note – this is a different page than the original, but keeps all the same content. We track lead sources and use different landing pages for traffic from different sources)
3. What is the traffic source? Organic, direct, PPC, etc.?
Cold traffic from a dating website – CPM. We tested a lot of different bids and were very surprised about the most profitable bid price.
4. Which part of page did you select for the test and what variations did you test?
We initially ran a test with three different variations each of headline and sub-head. Once we found the best combination for those, we split the “Submit” button on a whim. One said “Free Instant Access” and one said “Watch The Video”. See images below:
Version A: Get Instant Access (11.9% conversion rate)
Version B: Watch the Video (15.3% conversion rate)
Editor’s note: so now you know that ‘Watch the video‘ increased conversions from 11.9% to 15.3%. A total increase of 28%!
5. Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Pushing for the action of “Watch The Video”. The subhead that won (“Get The Free ‘Talk Like A Ten’ Video Now!!!”) also mentioned that they could “get” the “video”.
6. What results did you get? Were you surprised by the results?
15% opt-ins for cold traffic off this particular site is amazing – We’re running ads/offers into a few very nichey demographics on that site that generate huge CTR’s (.300-.400; in comparison, the CTR on this campaign is .086 for our best creative) and haven’t beaten this conversion for any other squeeze page.
7. Any lessons which can be derived from your test?
I’m reminded of the parable where an old friend of Henry Ford’s said “Henry, why don’t you ever buy any bolts from me?” to which Mr. Ford replied “Heck Joe, you’ve never asked!” It’s always worth including action verbs like “get”, “watch”, “download”, “enter” etc. in tests.
8. How valuable was Visual Website Optimizer for this test?
It’s so much easier – and more powerful – than Google’s website optimizer. We’re finding it to be an indispensable tool that made me excited about running tests again.
9. Short background information about your business.
The Social Man is in business is to make men awesome. We have a variety of products and coaching packages that cover everything from flirting over text messaging, to how to be sexier, to overcoming social anxiety. I previously ran an enterprise software company, and feel very blessed to be doing something I love that’s employs passionate people and has helped so many men. My business, Turnseven, Inc., manage a few other niche brands in addition to The Social Man.
Boy, what a year 2010 has been for A/B testing industry and Visual Website Optimizer. We saw tens and hundreds of small/medium sized businesses increase sales and conversions on their websites. Our repository of A/B testing case studies is the proof that A/B testing works! To celebrate the excitement of 2011, I have complied the list of our best A/B testing case studies in 2010 (in order of bestness). Hope they will inspire you to kick-start A/B testing campaign in your organization in 2011.
Original Landing Page (with embedded signup form)
Variation (no signup form): 60% increase in signups!
Who would have thought that removing a signup form from signup page can actually increase conversions by 60%? Vendio, an Alibaba.com company, used Visual Website Optimizer to do an A/B test on their signup landing page. In one version, they removed the signup form altogether and instead wrote about benefits of signing up. The end result was that the version with no signup form eventually saw 60% more signups. Click here to read the full case study (includes screenshots)
Control: 14.5% conversion rate
Variation: 28% improvement conversion rate!
Soocial tested multiple variations of their sign up button in order to increase clickthroughs to the registration form. Variations included call-to-actions such as “Sign up for free”, “Free sign up”, etc. The A/B test results revealed that one variation increased conversions dramatically. Click here to read the full case study (includes screenshots)
Control (paintings)
Variation (photos): 95% increase in CTR!
In two separate A/B tests, it was found out that human face photos doubled the conversion rate on a website. The result was striking because the tests were conducted by two completely unrelated companies, yet they arrived on the same conclusion. Click here to read the full case study (includes screenshots)
Original (control)
Redesigned (variation): 85% increase in downloads!
You Need A Budget makes easy-to-use personal financial management software. They split tested their product tour page by including more screenshots, adding a prominent testimonial and reducing text clutter. The redesigned page increased downloads of the free trial by 85%. Click here to read the full case study (includes screenshots)
Original version
Redesigned version: 20% increase in sales!
AquaSoft redesigned their sales page by giving it a modern, clean look. They also added trust building elements such as money-back guarantee, assurance seal, etc. Using Visual Website Optimizer, they effortlessly tested old v/s new sales page. After multiple phases of testing, they successfully increased total sales by 20%. Click here to read the full case study (includes screenshots)
Well, we already have a lineup of many interesting split testing case studies. From product pages to eCommerce sites. From effect of video on signups to effect of number of steps in checkout process on sales. We have a case study lined up for all these situations.
To keep updated with our latest A/B testing case studies, subscribe blog updates through email (powered by Feedburner).
Let’s admit it: your boss has certain opinions on the website design, the usability consultant has some “best-practices” suggestions and the sales team wants specific changes to increase leads. Heck, even your office receptionist thinks that you could add Facebook and Twitter buttons on the homepage. Needless to say, everyone in your office has an opinion on website design and if you are the one with Photoshop/HTML skills, it can drive you crazy fairly quickly. So, how to cope up with suggestion/feedback overload all the while trying to increase conversion rate on the website?
Rules of the game are pretty simple. You declare a split testing tournament at the Friday office lunch to win a booty of say $100 (or any other motivating prize). Anyone in your office (or team) with an opinion on site design gets to create a variation for a split test. You then pit all variations against each other in a live split test and quite obviously the person who came up with winning variation wins the prize booty!
This strategy brings out the best talent from across the team and helps you test diverse ideas at once. Since inputs for improving conversion rate and sales comes from different team members, this also helps you avoid designer’s block (of coming up with highly converting ideas and designs).
Visual Website Optimizer is the perfect tool for playing this game. Here’s why:
Do you think this is a killer idea? Have you ever tried this strategy in your office? At Wingify, we keep playing this game (after all, we make split testing tool). You also try this and let us know your thoughts and opinions. We have heard that the split testing game works like wonder to increase conversion rate and sales.
Let’s admit it: getting ideas for A/B and multivariate tests is a hard job! It becomes even harder because of your familiarity with your website or landing page. Since you breathe-and-eat your website every day and know all its intricacies, it becomes incredibly hard to think of improvements. This phenomenon even has a medical name: Blind Spot.
To get ideas on what you can test in your next A/B or multivariate test, we have a A/B testing case study search engine called Ideafox. Also, earlier on this blog I talked about a service called Feedback Army which lets you have feedback from 10 people for $10 (I see the price is $15 now). Feedback Army is great service and we had received lots of relevant feedback from that site (in addition to some actionable ideas for A/B testing). As also noted in original post, the only drawback of this service is that people who give feedback are hired from Mechanical Turk, so feedback length is short and doesn’t go into too much depth (since those people are only paid 50 cents per feedback). Inspite of this, Feedback Army has a great ROI for an investment of $15.
What if you need feedback on your website from a professional?
Enter Concept Feedback, the website to get expert website evaluations and feedback. It is a community of design, usability and conversion lovers who comment and give feedback on your website and landing page. They have experts for three categories of feedback:
Experts have years of experience in their respective fields along with professional degrees in design. So, you can be guaranteed to get top-notch feedback on your website from Concept Feedback. Of course, all this comes at a price. You need to shell out $100 per expert feedback. So if you want feedback from 5 experts, that is going to cost you $500. It’s a little expensive but as you will read below, the quality and amount of feedback that you get from an expert has way more value than $100.
Evaluation of Visual Website Optimizer website on Concept Feedback
To try out the service, I evaluated Concept Feedback to get feedback on Visual Website Optimizer website design. Oh boy, I was pleasantly surprised with the quality of feedback on as many five different pages of our site: homepage, signup page, features page, case study page and blog. Here is some of the feedback produced verbatim:
All these comments come with a screenshot of your page so you can see where exactly the expert is referring to when he says “make your headlines sans-serif” (in this case, he was talking about our blog).
What’s more? The expert even quickly created a mockup of what according to him our site should look like. Here is the version he redesigned:
I think this redesign is definitely an improvement over existing design. What do you think?
Conclusion
For getting ideas for A/B and multivariate tests, nothing works better than feedback and evaluation services such as Feedback Army and Concept Feedback. The ‘strategy’ feedback type in Concept Feedback is especially relevant to A/B tests because there you have access to experts who have experience and knowledge in conversion rate optimization.
So, next time you are stuck and can’t think of anything to improve on your site, you know where to get feedback from.
Visual Website Optimizer has a feature that allows running of same A/B or multivariate test on multiple pages at once. For example, if your site has several product categories and you would like to run a test only on one category, all you need to do is to enter a URL pattern like http://example.com/shoes/* in Visual Website Optimizer.
The URL pattern above will make the test run on all pages in the shoes directory. This feature is very handy because you can test common call-to-actions, headers, footers, sidebars, etc. consistently on multiple pages at one go. What’s more, a consistent variation is shown on all the pages in a test. So if a user sees red variation on one shoes page, he will always see the same red variation on all other shoes pages.
Sometimes (primarily due to SEO reasons) there is no clear pattern in the URLs. That is, it may be possible that there are several independent URLs for category or product pages without any apparent pattern. For example, if there are URLs such as http://example.com/adidas/, http://example.com/reebok/, http://example.com/microsoft/, http://example.com/adobe/, etc. and you wish to run the test on only on shoes pages (the first two URLs), how are you going to do that? Until now it wasn’t possible to setup such a test, but with our new feature Custom URLs it is very simple to setup such tests.
Actually, there are three easy steps if you want to use custom URLs:
On all pages where you need to run a specific test, output _vis_opt_url variable and modify VWO code as following (bold parts are extra modifications):
<head> ... // Only on pages where this test is to be run var _vis_opt_url = 'http://example.com/my_shoes_pages'; ... ... ... // Common VWO Code var _vis_opt_url = typeof(_vis_opt_url)=="undefined" ? document.URL : _vis_opt_url; .... '&url='+encodeURIComponent(_vis_opt_url) +'&random='+Math.random() ... </head>
We do realize that the implementation is bit complex but we wanted to roll out the feature to all customers (many enterprise customers have been using it already). We are here to help you implement VWO code for custom URLs. Simply contact support@wingify.com if you need our help and assistance.
Many times A/B testing is not limited to a single conversion goal. In fact, your test variations usually affect many different conversion goals on your site such as free trial signups, paid signups, newsletter subscription, etc. Measuring all these goals for a test is important because a variation may work brilliantly for one goal (e.g. it increases free signups) but may perform worse for your other goals (e.g. decreases your paid signups). If you don’t measure multiple goals for your A/B test, you are essentially flying blind in the dark and may end up making wrong decisions (based on a single conversion goal).
Case Study
A brilliant example of this is a recent A/B test by one of our customers: Guido Jansen (a Magento specialist). He runs Dutchento.org which is the official Dutch community for Magento CMS. The goal of his recent A/B test was to increase subscriptions for Dutchento’s newsletter and RSS subscription. The call-to-action for subscription is located in a box on all pages of the site. See the control version of subscription box below and note that there is no explicit incentive for a visitor to subscribe:
Guido tested a variation of this subscription box which included a title and some benefits in a bullet list. Here is his hypothesis for increasing subscriptions:
People don’t just subscribe to a newsletter or newsfeed for nothing, you should convince them it has added value above just visiting the website. So what I wanted to test is if adding convincing reasons to subscribe would increase the newsletter and newsfeed subscription rate. I measured the impact of the convincing reasons on clicks on both the newsletter and newsfeed links.
Here is how his variation (of subscription box) looks like:
Results
As expected, Guido saw a significant improvement of the newsletter click rate (+190.31%). However, newsfeed click rate decreased (-44.46%) which did surprise him (and us!). He expected that the convincing reasons would affect both positively, but apparently it had a negative effect on newsfeed clicks.
The reason why clicks on newsfeed decreased is not clear but we believe that the benefits in variation were so compelling that visitors chose to get the blog updates via email (where they will be sure to read them) rather than RSS reader (where they may miss them). A great way to get more insight into this would be to randomize the position of newsfeed / newsbrief to eliminate the positional effect of those links.
Testimonial
Guido used Visual Website Optimizer for A/B testing and here is what he had to say:
Visual Website Optimizer was very valuable [for testing]. It’s the easiest A/B and Multivariate testing tool I know. It’s great not to be dependent on the development department to create and run your tests.
Key Lesson
Whatever be the actual reason of decreased clicks on RSS feed, one key lesson is jumps out of the case study: always measure multiple conversion goals in your test. Relying on a single conversion goal hides a lot of valuable information from you. So, make sure you add multiple goals to your next A/B test.
Do you have more examples of using multiple goals in A/B test? What are your thoughts on this case study? Please let us know by leaving a comment below – will be happy to discuss!
A lot of our potential customers are concerned whether A/B or multivariate testing will have an effect on their search engine rankings. They (obviously) aren’t looking to improve their rankings by using A/B testing, rather their concerns relate to the myth that testing may negatively impact search engine optimization efforts.
Broadly, there mainly two major concerns:
Why A/B testing is not content cloaking
Cloaking content was all rage back in the early days of search engines (think 1990s). In those days, SEO was all about keyword stuffing. So, smart SEO geeks used to display a page full of keywords when a search engine bot visited to crawl / index the page. While if a user visited the same page, they displayed the default (normal) version. This strategy of keyword stuffing used to work like wonders so naturally search engines started devising clever ways to detect and penalize such cloaking and it may appear that pages where A/B testing is on will also get penalized because it is kind of cloaking.
Good news is that thanks to Google’s PageRank algorithm, keyword stuffing no longer works. So, there is little incentive for search engines to penalize content cloaking. Moreover, unlike yesteryears’ static HTML sites, today’s search engines have come to expect highly dynamic AJAX driven sites. So, they no longer consider swapping content dynamically as cloaking. If you use a multivariate testing software to swap different parts of your page, it is not cloaking! Also, all the major search engines (Google, Yahoo, Bing, etc.) are well versed with concept of A/B split testing and they do many such tests on their own sites every single day. You wouldn’t expect search engines to penalize others for doing the same activity that they themselves do regularly, would you?
Key lesson: Doing A/B testing by dynamically swapping elements on page (using JavaScript) is NOT content cloaking. It is absolutely safe and should not have any negative impact on Search Engine Optimization efforts.
Split URL testing (redirecting traffic to multiple versions) and the issue of content duplication
As I mentioned above, this is another issue that worries many prospective customers. While one approach for split testing is to dynamically swap page elements (such as headlines, images, text, etc.), another approach is to host different variations of page on your website. So, if you want to split test product.html, you will create variation pages and host them as product1.html, product2.html, etc. A split testing software will redirect all traffic coming to product.html to the variation pages. Since product1.html and product2.html will have similar content as product.html, it worries some people that search engines will consider them duplicate content and penalize rankings.
An important point to note here is that search engines only penalize if you steal or host content from a different domain; here all variations pages host on your own site and you OWN that content and are free to do with it whatever you want to. Many dynamic websites (shopping carts, directories, etc.) today host the same content in different formats. Search engines don’t penalize them because that unique content is only found on their domain and no where else. Same is the case with your split testing URLs. You don’t get negatively impacted by it because you haven’t stolen that content — it is all yours!
A more important point is answer to this question: how will search engines come to know that there are different variations of your main page? Your variation pages can only be reached if there is a link pointing to it. Since you don’t link to your variation pages from anywhere from the site or else-where, search engines won’t know that those pages even exist. For the truly paranoid, you can instruct search engines not to index you variations pages. You can do it either using no-index meta attribute or by using robots.txt
Lesson: A/B split testing has absolutely NO negative impact on your SEO efforts!
Rest assured, you can use Visual Website Optimizer without worrying about its impact on search engine rankings. If you have a specific SEO and A/B testing related question, let me know in comments below.
