One of the questions we often hear from users who are new to A/B testing is what should they test. This does seem like a simple question but it can be quite baffling to the new-comers. This post is one such attempt to narrow down certain options to test, the changes that worked for others might work for you. The change here that we are talking about is testing having a video on homepage or product page. Previous case studies prove that it usually works wonderfully. One such case study is video increases conversion rate of a landing page by 46% and in another one they tested two call to actions ‘Watch the video’ v/s ‘Get Instant Access’ and found out that the video one increased newsletter signups by 28%.
In this latest case study, one of our clients Buy Real Twitter Followers made a variation with a small video explaining about their service on the homepage. This little change helped them increase their sales by 216%. Buy Real Twitter Followers was formed as a response to the need made by the growing Twitter advertising market. Getting Twitter followers has been a large concern by many network marketers and businesses whose primary aim is to advertise through Twitter.
Visitors usually tend to skip the blocks of text. This case study only goes on to prove that users much rather prefer to see the video for the product validation than reading block of text to get the gist of the product.
Original Page
Variation Page
This is what Rene Harvey had to say about the A/B testing tool they used: Visual Website Optimizer: “VWO was very helpful, it was very easy to use and provides a room for designers such as myself to customize without having to change the original code.”
As always, everything is open to testing and the idea is to find what change benefits the most. Let us know what you think about this case study in the comments below.
Starting with a new series on this blog called Revisiting the Classics, I will re-look at our A/B testing case studies and throw some new light on it (with additional commentary). The case study that I picked for this post relates to human photos on landing pages and whether they increase conversion rate. Before reading rest of the post, make a guess: do human photos (male/female) online really have any subconscious effect on visitors?
There has been significant scientific research on this topic particularly for eCommerce websites. When you are considering a purchase decision, the main issue online is of trust. How do you know if a particular website is trust worthy? Taking lessons from TV advertising and general marketing principles, people running eCommerce websites think that associating photos of people with products engenders trust. Of course, human photos = increased trust is just a hypothesis. But is it really true?
To answer this, I analyzed several research papers (which are published in peer-reviewed scientific journals). Following are some of the main ones (along with excerpts):
Not just scientific research, real world A/B tests that I have seen also corroborate the hypothesis of human photos may increase conversion rate. Two of our customers used our A/B testing tool (Visual Website Optimizer) to test presence of human photos and its impact on conversions. Following sections detail what they did and what they found out.
Medalia Art sells Brazilian and Caribbean art online. Since they are an online art shop, they showcase paintings from famous artists on their homepage. They decided to test what happens if paintings are replaced by the artists’ photos.
The goal for this A/B test was to increase visitor engagement (defined as a click on any link on the homepage; it is the inverse of the bounce rate). They didn’t use sales as a conversion rate as many of their sales happen after interaction on the telephone with the customer.
Here are the screenshots of the control (paintings) and the winning version (photos).
- Original (with paintings)-
- Variation (with photos)-
Results?: Medalia found out that with paintings the conversion rate was 8.8% but if paintings are replaced with artist photos, the conversion rate increases to whopping 17.2%. That’s an increase in conversion rate of more than 95%! (results were statistically significant) How cool is that for doing a slight change?
Another user, Jason Thompson, conducted an A/B test on his blog to see if replacing contact icon with his own photo would lead to more people contacting him. Following is screenshot of original and variation:
|
Original (with icon) |
Variation (with photo) |
 |
 |
|
3.7% conversion rate |
5.5% conversion rate |
Results clearly show that the version with Jason’s photo had 48% more conversions as compared to generic icon. (Again, results are statistically significant).
Jason commented on the test results:
People want to connect with other people emotionally, the photo makes that emotional connection so much easier and as the test is proving, drives people to the contact form more than a nondescript icon.
There are hundreds of more research studies and probably more A/B test results that you can read but from what I have read and observed, here are some salient points about human photos on websites:
Hope you found this article useful. If you have any comments or suggestions, please let me know.
Recently on Hacker News, someone commented that A/B testing has become like snake oil, making grand promises to be a panacea for increasing conversion rates. This comment troubled me not because the commenter was wrong (he was!) but because how certain people view A/B testing. Let me put this straight and clear:
In fact, we recently featured a guest post by Noah of Appsumo who revealed that only 1 out of 8 split tests that they run produce any actionable results. Yep, read that once again: only 1 out of 8 split tests work. Snake oil guarantees to be a cure for all diseases; A/B testing has no such guarantees.
I think the reason why some people would view split testing suspiciously is because we release a lot of A/B testing case studies. In fact, with a user base so large (6000+ at the time of writing), we see many of our customers getting good results all the time and they if want to share those results with the world, we publish them on our blog.
So, when one week we publish a case study titled 3 dead-simple changes increased sales by 15% and the next week we publish another case study titled How WriteWork.com increased sales by 50%, it naturally makes some people assume using Visual Website Optimizer to do A/B testing would ensure a growth in sales. No, using Visual Website Optimizer won’t guarantee anything. It is just an A/B testing tool (albeit, a very good one!) but the real work done is always by the craftsman, not the tool s/he uses.
Most success stories or case studies (be it about A/B testing, getting rich or losing weight) that you read about on the Internet suffer from something known in psychology as survivorship bias. In context of A/B testing, what it means is simple:
If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.
Obviously, 99 case studies about how A/B testing did not produce any results in unexciting and will bore our blog readers. Hence, this post is a short reminder about all those case studies that don’t produce results.
The reason we publish A/B testing case studies is to show potential of what can be achieved using A/B testing. It is similar to holding an art exhibition for Picasso paintings which inspires people to appreciate the work and perhaps pick up brush themselves to start painting hoping to achieve level of success that Picasso achieved. To reiterate, just as picking up a brush and canvas does not guarantee good art, doing A/B testing does not guarantee increase in sales.
In fact, specifically to A/B tests, replicating them almost never works. So, if you read a case study of how video increases conversions by 46% and then implement it on your site hoping for the same magical increase, it may or may not happen. What worked for them may not work for you! So, why publish case studies at all? Here’s why:
A/B testing case studies give ideas on what you can test on your website
Yep, that’s all there is to A/B testing case studies.
And, yes, I hope I convinced you that A/B testing is not snake oil!
Through his website SixPackAbsExercises.com, personal trainer Carl Juneau teaches men the best ab exercises for getting six pack abs. Carl heard his top competitor doubled his sales after he started to use video to market and demonstrate his products so he decided to use Visual Website Optimizer to test video on this website. Carl carried out three split tests which conclusively proved that videos increased conversions on his website by as much as 46%.
In first split test, he tested two variations of the “call to action” on his sales page.
Variation 1 of Carl’s sales page. Call to action: “Next Page Watch Video Preview.”
Here are the results for this test (goal being click on call-to-action and go to next step in funnel). Both calls to action that hinted at a video significantly increased the number of visitors who clicked to the sales page (step 3). The best-performing variation (variation 1) increased conversion by 14.18%.
In second split test, Carl tested one variation of the sales page:
Here are the results for test #2 (goal was to go to next step of funnel). The video sales page significantly increased the number of visitors who clicked to the price/guarantee page by 46.15%.
Third split test was a follow-up test. In it, Carl tested:
To his surprise, the video-only page won. Here are the results. Adding the long salesletter below the winning sales video significantly reduced the number of visitors who clicked to the price/guarantee page by 35%.
Carl couldn’t explain these results. He shared his best guess:
I’m guessing visitors were intrigued by the sales video and clicked through to the price/guarantee page to get more info. They may have been turned off by the long salesletter when I added it to the video and lost the excitement created by the short, punchy video.
Video converts! It did so when mentioned in a “call to action” (a 14.18% increase) and also when used to sell (35% and 46.15% increases in two different tests).
As more and more surfers have broadband Internet and powerful computers, video seems to be slowly taking over the web. Carl increased his conversions with video… why not test it and see if it increases yours?
Carl also submitted feedback on his experience with Visual Website Optimizer:
I carried out these test using Visual Website Optimizer. Before switching to Visual Website Optimizer, I had one solid year of testing under my belt (63 tests) with Google Website Optimizer.
Between the two, Visual Website Optimizer wins hands down. It’s easier to use, faster, and more user-friendly.
When you signup for a 30 day free trial (or purchase a subscription), you install a piece of code on each of your test page. Once the code is installed, you can perform as many tests as you want. You don’t need to re-install a new piece of code for every new test. This was the case in Google Website Optimizer, and it made testing cumbersome.
Visual Website Optimizer’s website itself is faster. It loads in a flash.
Creating tests is intuitive, quick, and easy. It’s done through a “visual” interface. In short, Visual Website Optimizer loads the page you want to test. You then select the sections on your page you want to test. For each section, you specify different versions using the what-you-see-is-what-you-get editor (FrontPage style). You also have the option to edit the HTML directly. When you’re done, you can preview each version before you start the test. It’s that easy, and results are available right away!
One last word on customer support: in two months, I’ve emailed the team 26 times. On average, I’ve gotten a response the same day. VWO team is smart and dedicated. This level of customer support is unheard of.
As you might guess, I strongly endorse this service.
Full disclosure: I’m in no way linked to Visual Website Optimizer, except that I use the software. I’m not being paid for this review.
Carl Juneau
http://www.sixpackabsexercises.com/
Editor’s note: we had a similar case study last week where the call to action ‘Watch the video’ increased conversions by 28%. These two case studies make a strong case for testing a video on your website. It may probably do wonders to your conversions!
Last week we ran an A/B test contest where you had to guess which button increased newsletter signups by 28%. Was it ‘watch a video’ or was it ‘get instant access‘? We got a total of 51 submissions (tweets + blog comments) and would you believe it: 51% said ‘Get Instant Access‘ would win while 49% said ‘Watch a video‘ will win. This is incredible and clearly demonstrates how hard is it to guess what is going to work on your website!
Before announcing the correct answer, we’d like to announce the winner of $3000 worth Visual Website Optimizer subscription. Please join me for the drum roll… and the winner of this contest is: Stephan Schubert and here is the winning tweet. Congratulations, Stephan!
I recently interviewed The Social Man (one of the Visual Website Optimizer customers) on their successful split test. As you will read, this test is typical example of how a single change (in call-to-action) can result in significant increase in conversions. Almost all our case studies make the same point about A/B testing which is worth repeating twice: (seemingly) small changes can increase conversions on a landing page. So, you should always be testing.
Here is the full interview:
1. What was the conversion goal of the split test?
To convert visitors to email newsletter subscribers, and to get them to see a video about how a guy can learn to “talk like a ten”.
2. On which page did you run the test?
http://www.becomeunbreakable.com/1/talk-like-a-ten/indexVWO.php
(Please note – this is a different page than the original, but keeps all the same content. We track lead sources and use different landing pages for traffic from different sources)
3. What is the traffic source? Organic, direct, PPC, etc.?
Cold traffic from a dating website – CPM. We tested a lot of different bids and were very surprised about the most profitable bid price.
4. Which part of page did you select for the test and what variations did you test?
We initially ran a test with three different variations each of headline and sub-head. Once we found the best combination for those, we split the “Submit” button on a whim. One said “Free Instant Access” and one said “Watch The Video”. See images below:
Version A: Get Instant Access (11.9% conversion rate)
Version B: Watch the Video (15.3% conversion rate)
Editor’s note: so now you know that ‘Watch the video‘ increased conversions from 11.9% to 15.3%. A total increase of 28%!
5. Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Pushing for the action of “Watch The Video”. The subhead that won (“Get The Free ‘Talk Like A Ten’ Video Now!!!”) also mentioned that they could “get” the “video”.
6. What results did you get? Were you surprised by the results?
15% opt-ins for cold traffic off this particular site is amazing – We’re running ads/offers into a few very nichey demographics on that site that generate huge CTR’s (.300-.400; in comparison, the CTR on this campaign is .086 for our best creative) and haven’t beaten this conversion for any other squeeze page.
7. Any lessons which can be derived from your test?
I’m reminded of the parable where an old friend of Henry Ford’s said “Henry, why don’t you ever buy any bolts from me?” to which Mr. Ford replied “Heck Joe, you’ve never asked!” It’s always worth including action verbs like “get”, “watch”, “download”, “enter” etc. in tests.
8. How valuable was Visual Website Optimizer for this test?
It’s so much easier – and more powerful – than Google’s website optimizer. We’re finding it to be an indispensable tool that made me excited about running tests again.
9. Short background information about your business.
The Social Man is in business is to make men awesome. We have a variety of products and coaching packages that cover everything from flirting over text messaging, to how to be sexier, to overcoming social anxiety. I previously ran an enterprise software company, and feel very blessed to be doing something I love that’s employs passionate people and has helped so many men. My business, Turnseven, Inc., manage a few other niche brands in addition to The Social Man.
Boy, what a year 2010 has been for A/B testing industry and Visual Website Optimizer. We saw tens and hundreds of small/medium sized businesses increase sales and conversions on their websites. Our repository of A/B testing case studies is the proof that A/B testing works! To celebrate the excitement of 2011, I have complied the list of our best A/B testing case studies in 2010 (in order of bestness). Hope they will inspire you to kick-start A/B testing campaign in your organization in 2011.
Original Landing Page (with embedded signup form)
Variation (no signup form): 60% increase in signups!
Who would have thought that removing a signup form from signup page can actually increase conversions by 60%? Vendio, an Alibaba.com company, used Visual Website Optimizer to do an A/B test on their signup landing page. In one version, they removed the signup form altogether and instead wrote about benefits of signing up. The end result was that the version with no signup form eventually saw 60% more signups. Click here to read the full case study (includes screenshots)
Control: 14.5% conversion rate
Variation: 28% improvement conversion rate!
Soocial tested multiple variations of their sign up button in order to increase clickthroughs to the registration form. Variations included call-to-actions such as “Sign up for free”, “Free sign up”, etc. The A/B test results revealed that one variation increased conversions dramatically. Click here to read the full case study (includes screenshots)
Control (paintings)
Variation (photos): 95% increase in CTR!
In two separate A/B tests, it was found out that human face photos doubled the conversion rate on a website. The result was striking because the tests were conducted by two completely unrelated companies, yet they arrived on the same conclusion. Click here to read the full case study (includes screenshots)
Original (control)
Redesigned (variation): 85% increase in downloads!
You Need A Budget makes easy-to-use personal financial management software. They split tested their product tour page by including more screenshots, adding a prominent testimonial and reducing text clutter. The redesigned page increased downloads of the free trial by 85%. Click here to read the full case study (includes screenshots)
Original version
Redesigned version: 20% increase in sales!
AquaSoft redesigned their sales page by giving it a modern, clean look. They also added trust building elements such as money-back guarantee, assurance seal, etc. Using Visual Website Optimizer, they effortlessly tested old v/s new sales page. After multiple phases of testing, they successfully increased total sales by 20%. Click here to read the full case study (includes screenshots)
Well, we already have a lineup of many interesting split testing case studies. From product pages to eCommerce sites. From effect of video on signups to effect of number of steps in checkout process on sales. We have a case study lined up for all these situations.
To keep updated with our latest A/B testing case studies, subscribe blog updates through email (powered by Feedburner).
Many people seem to get confused about the terms A/B testing, split testing and multivariate testing. Essentially, A/B testing and split testing are the same concepts but multivariate testing is different. So for all practical concepts, consider A/B Testing = Split Testing. The following table will illustrate difference between A/B testing (or split testing) and Multivariate Testing:
| A/B Testing | Multivariate Testing | |
| What is it? |
In A/B testing you split traffic amongst two or more completely different versions of a webpage (landing page, home page, etc.) The variations of your original page can differ in any manner. You can either just change the headline; or you can even change entire design, layout, offer and what not in the variations. What you change in A/B test is only limited by your creativity. For example, if you are A/B testing on your landing page you may want to create one version with 15% discount, the other with free shipping and one with the same offer but different page design. |
In multivariate test, you identify a few key areas/sections of a page and then create variations for those sections specifically (as opposed to creating variations of whole page in an A/B split test). So for example, in multivariate test you can choose to create different variations for 2 different sections: headline and image. A multivariate testing software will combine all these section specific variations to generate unique versions of page to be tested and then simply split traffic amongst those versions. That is, you will get these variations of the page: Headline1_Image1, Headline1_Image2, Headline2_Image1, Headline2_Image2. |
| How it works? |  |
 |
| Traffic Required |
Relatively Less |
Huge |
| Best used for |
Testing radically different ideas for conversion rate optimization |
Optimizing and refining an existing landing page or homepage without doing significant investment in redesign |
| Example Case Study |
Signups increased by 60% after actually removing the signup form |
I hope that was a good comparison between A/B testing and multivariate testing. If you think I left any point, please let me know. I will update the comparison table accordingly.
Last year, at the dawn of 2010 I wrote four reasons why 2010 is going to be a year of A/B split testing. Now that 2010 is coming to a close, I thought it will be great to revisit what happened in our tiny little (yet growing) industry of split testing and conversion rate optimization. Undoubtedly, the year saw explosive growth in the industry (and also for our A/B testing tool – Visual Website Optimizer). It was incredibly hard to pick 10 best A/B testing resources (not because there weren’t any, but rather there were simply too many great tools, websites, forums, tutorials, etc.). Anyhow, I managed to compile following is the list of top 10 resources (in no particular order):
WhichTestWon.com is an interesting website which features a new A/B split test every week. You can vote and discuss about which variation won in the test. It is a fun way to keep learning what usually works for increasing conversion rate on website.
ABTests.com is an open forum where anyone can share and discuss their A/B test results. ABTests.com and WhichTestWon.com are two great websites when it comes to reading real-life A/B testing case studies. You simply can’t get enough of them!
The Ultimate Guide To A/B Testing is a guest post I wrote for Smashing Magazine. I had tried to make this post as the definite resource for anyone who wants to get started with A/B testing. Judging from #2 position on Google for ‘A/B testing’ (#1 is Wikipedia), I think I have achieved my goal with that guest post.
#crochat is a hashtag used by a group of conversion rate optimization enthusiasts to chat on this topic every Thursday. I have seen everyone from experts to vendors to beginners participating in the chat and discussing interesting topics related to A/B testing.
MarketingExperiments.com is an organization that does numerous split and multivariate tests every week and have been doing so for years now. They have built an impressive research library which details how they improved website conversions and sales on different kinds of websites: eCommerce, publishers, lead-gen landing pages, etc.
Conversion Conference is the first and only conference dedicated to the art of conversion rate optimization. The next edition will be held in San Francisco in March, 2011.
WhichMVT.com is a site where you can compare different multivariate testing tools (including Visual Website Optimizer). You can also read or leave review of different tools. (Just in case you happen to use VWO and love it, please take 5 minutes out to leave a review here).
Online Behavior is a new magazine by industry experts which features columns on behavioral targeting, web analytics and testing & optimization. All articles are of top-notch quality. If you are interested in conversion rate optimization, subscribing to their feed is a must.
GetElastic.com exclusively focuses on conversion rate optimization and a/b split testing for eCommerce websites. It’s a must read blog for anyone even remotely related to an eCommerce business.
You knew this was coming, didn’t you? 
I am very happy that Visual Website Optimizer Blog has seen explosive growth in readership and reach in last couple of months. This blog is not just about product/feature announcements only. Rather majority of posts relate to A/B testing case studies and conversion rate optimization how-tos. All time hits include: landing page optimization tips, impact of SEO on A/B testing, top 10 eCommerce sites (by conversion rate), etc.
Are there any other A/B testing resources that I missed? Please leave a comment below!
It is interesting how counter-intuitive A/B testing results can get. One of the best examples of that counter-intuitiveness is a recent A/B split test. This test was done by a Visual Website Optimizer customer Vendio, an Alibaba.com company that specializes in providing free e-Commerce stores to its merchants. They have special landing pages for the free store signup and the objective of this test was to increase signup conversion rate on one of those landing pages.
Their original landing page (control) combined marketing content and registration fields in an attempt to reduce the number of clicks for a successful registration. This is how it looked:
Original Landing Page (with embedded signup form)
Note that it uses the so-called “best-practice” of embedding the signup form in the landing page itself. They had long been using the layout of the original page because as a best practice they presumed that reducing the number of clicks for a registration increased conversion rate. Although the page was performing relatively well, they wanted to make sure the included registration fields weren’t too aggressive or limiting in any manner.
So, the variation that they tested had somewhat similar imagery and content but the page didn’t include any registration fields and had slightly different styling. Clicking on ‘Signup Now’ button simply took the visitor to a page with signup form. In other words, Vendio added an extra step in their conversion funnel. Not a smart move, huh? This is how variation looked like:
Variation (no signup form) – 60% increase in signups!
Guess what? The page without the registration fields performed better – much better – to the tune of a 60% increase in conversions! Here is what Vendio had to say about the results:
Best practices are NOT always true! It’s still hard to believe, but the numbers don’t lie.
So, that’s the biggest lesson here and it is worth repeating: “Best practices are NOT always true”. If you do changes on your website or landing pages without A/B testing them, you are actually flying in the dark. Another lesson here is that it is it is worth testing radically different ideas – which, on the first glance, may appear not-so-smart (like removing signup form from the landing page!).
After the successful test, Vendio gave us a great testimonial:
Visual Website Optimizer was extraordinarily valuable. It was far easier to use than other solutions (some of which we couldn’t even get to work properly after implementation). Without VWO we would still be missing out on all those registrations, and we are continuing to see great and other surprising results with VWO.
Let’s admit it: getting ideas for A/B and multivariate tests is a hard job! It becomes even harder because of your familiarity with your website or landing page. Since you breathe-and-eat your website every day and know all its intricacies, it becomes incredibly hard to think of improvements. This phenomenon even has a medical name: Blind Spot.
To get ideas on what you can test in your next A/B or multivariate test, we have a A/B testing case study search engine called Ideafox. Also, earlier on this blog I talked about a service called Feedback Army which lets you have feedback from 10 people for $10 (I see the price is $15 now). Feedback Army is great service and we had received lots of relevant feedback from that site (in addition to some actionable ideas for A/B testing). As also noted in original post, the only drawback of this service is that people who give feedback are hired from Mechanical Turk, so feedback length is short and doesn’t go into too much depth (since those people are only paid 50 cents per feedback). Inspite of this, Feedback Army has a great ROI for an investment of $15.
What if you need feedback on your website from a professional?
Enter Concept Feedback, the website to get expert website evaluations and feedback. It is a community of design, usability and conversion lovers who comment and give feedback on your website and landing page. They have experts for three categories of feedback:
Experts have years of experience in their respective fields along with professional degrees in design. So, you can be guaranteed to get top-notch feedback on your website from Concept Feedback. Of course, all this comes at a price. You need to shell out $100 per expert feedback. So if you want feedback from 5 experts, that is going to cost you $500. It’s a little expensive but as you will read below, the quality and amount of feedback that you get from an expert has way more value than $100.
Evaluation of Visual Website Optimizer website on Concept Feedback
To try out the service, I evaluated Concept Feedback to get feedback on Visual Website Optimizer website design. Oh boy, I was pleasantly surprised with the quality of feedback on as many five different pages of our site: homepage, signup page, features page, case study page and blog. Here is some of the feedback produced verbatim:
All these comments come with a screenshot of your page so you can see where exactly the expert is referring to when he says “make your headlines sans-serif” (in this case, he was talking about our blog).
What’s more? The expert even quickly created a mockup of what according to him our site should look like. Here is the version he redesigned:
I think this redesign is definitely an improvement over existing design. What do you think?
Conclusion
For getting ideas for A/B and multivariate tests, nothing works better than feedback and evaluation services such as Feedback Army and Concept Feedback. The ‘strategy’ feedback type in Concept Feedback is especially relevant to A/B tests because there you have access to experts who have experience and knowledge in conversion rate optimization.
So, next time you are stuck and can’t think of anything to improve on your site, you know where to get feedback from.
