Posted in A/B Split Testing, Case Studies, How To on January 18th, 2011
Through his website SixPackAbsExercises.com, personal trainer Carl Juneau teaches men the best ab exercises for getting six pack abs. Carl heard his top competitor doubled his sales after he started to use video to market and demonstrate his products so he decided to use Visual Website Optimizer to test video on this website. Carl carried out three split tests which conclusively proved that videos increased conversions on his website by as much as 46%.
A/B Test #1 (call-to-action test)
In first split test, he tested two variations of the “call to action” on his sales page.
- The control was: “Next Page Read Sample of Book”
- Variation 1 was: “Watch Video Preview”
- Variation 2 was: “Watch my #1 Abs Exercise On Video”
Variation 1 of Carl’s sales page. Call to action: “Next Page Watch Video Preview.”
Here are the results for this test (goal being click on call-to-action and go to next step in funnel). Both calls to action that hinted at a video significantly increased the number of visitors who clicked to the sales page (step 3). The best-performing variation (variation 1) increased conversion by 14.18%.
A/B Test #2 (salesletter test)
In second split test, Carl tested one variation of the sales page:
- The control was a long-form salesletter, which had been tweaked extensively over two years
- Variation 1 was a one-minute professionally produced video where Carl demonstrated his best abdominal exercise. At the end of the video, the “call to action” said that more exercises could be found in the manual and the workouts
Here are the results for test #2 (goal was to go to next step of funnel). The video sales page significantly increased the number of visitors who clicked to the price/guarantee page by 46.15%.
A/B Test #3 (follow up test)
Third split test was a follow-up test. In it, Carl tested:
- The winning video only
- The winning video at the top of the page plus his longstanding control salesletter below
To his surprise, the video-only page won. Here are the results. Adding the long salesletter below the winning sales video significantly reduced the number of visitors who clicked to the price/guarantee page by 35%.
Carl couldn’t explain these results. He shared his best guess:
I’m guessing visitors were intrigued by the sales video and clicked through to the price/guarantee page to get more info. They may have been turned off by the long salesletter when I added it to the video and lost the excitement created by the short, punchy video.
Lessons
Video converts! It did so when mentioned in a “call to action” (a 14.18% increase) and also when used to sell (35% and 46.15% increases in two different tests).
As more and more surfers have broadband Internet and powerful computers, video seems to be slowly taking over the web. Carl increased his conversions with video… why not test it and see if it increases yours?
Feedback on Visual Website Optimizer
Carl also submitted feedback on his experience with Visual Website Optimizer:
I carried out these test using Visual Website Optimizer. Before switching to Visual Website Optimizer, I had one solid year of testing under my belt (63 tests) with Google Website Optimizer.
Between the two, Visual Website Optimizer wins hands down. It’s easier to use, faster, and more user-friendly.
When you signup for a 30 day free trial (or purchase a subscription), you install a piece of code on each of your test page. Once the code is installed, you can perform as many tests as you want. You don’t need to re-install a new piece of code for every new test. This was the case in Google Website Optimizer, and it made testing cumbersome.
Visual Website Optimizer’s website itself is faster. It loads in a flash.
Creating tests is intuitive, quick, and easy. It’s done through a “visual” interface. In short, Visual Website Optimizer loads the page you want to test. You then select the sections on your page you want to test. For each section, you specify different versions using the what-you-see-is-what-you-get editor (FrontPage style). You also have the option to edit the HTML directly. When you’re done, you can preview each version before you start the test. It’s that easy, and results are available right away!
One last word on customer support: in two months, I’ve emailed the team 26 times. On average, I’ve gotten a response the same day. VWO team is smart and dedicated. This level of customer support is unheard of.
As you might guess, I strongly endorse this service.
Full disclosure: I’m in no way linked to Visual Website Optimizer, except that I use the software. I’m not being paid for this review.
Carl Juneau
http://www.sixpackabsexercises.com/
Editor’s note: we had a similar case study last week where the call to action ‘Watch the video’ increased conversions by 28%. These two case studies make a strong case for testing a video on your website. It may probably do wonders to your conversions!
Posted in A/B Split Testing, Case Studies on January 6th, 2011
Last week we ran an A/B test contest where you had to guess which button increased newsletter signups by 28%. Was it ‘watch a video’ or was it ‘get instant access‘? We got a total of 51 submissions (tweets + blog comments) and would you believe it: 51% said ‘Get Instant Access‘ would win while 49% said ‘Watch a video‘ will win. This is incredible and clearly demonstrates how hard is it to guess what is going to work on your website!
Before announcing the correct answer, we’d like to announce the winner of $3000 worth Visual Website Optimizer subscription. Please join me for the drum roll… and the winner of this contest is: Stephan Schubert and here is the winning tweet. Congratulations, Stephan!
‘Watch a video’ v/s ‘Get Instant Access’ – 28% increase in signups
I recently interviewed The Social Man (one of the Visual Website Optimizer customers) on their successful split test. As you will read, this test is typical example of how a single change (in call-to-action) can result in significant increase in conversions. Almost all our case studies make the same point about A/B testing which is worth repeating twice: (seemingly) small changes can increase conversions on a landing page. So, you should always be testing.
Here is the full interview:
1. What was the conversion goal of the split test?
To convert visitors to email newsletter subscribers, and to get them to see a video about how a guy can learn to “talk like a ten”.
2. On which page did you run the test?
http://www.becomeunbreakable.com/1/talk-like-a-ten/indexVWO.php
(Please note – this is a different page than the original, but keeps all the same content. We track lead sources and use different landing pages for traffic from different sources)
3. What is the traffic source? Organic, direct, PPC, etc.?
Cold traffic from a dating website – CPM. We tested a lot of different bids and were very surprised about the most profitable bid price.
4. Which part of page did you select for the test and what variations did you test?
We initially ran a test with three different variations each of headline and sub-head. Once we found the best combination for those, we split the “Submit” button on a whim. One said “Free Instant Access” and one said “Watch The Video”. See images below:
Version A: Get Instant Access (11.9% conversion rate)
Version B: Watch the Video (15.3% conversion rate)
Editor’s note: so now you know that ‘Watch the video‘ increased conversions from 11.9% to 15.3%. A total increase of 28%!
5. Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Pushing for the action of “Watch The Video”. The subhead that won (“Get The Free ‘Talk Like A Ten’ Video Now!!!”) also mentioned that they could “get” the “video”.
6. What results did you get? Were you surprised by the results?
15% opt-ins for cold traffic off this particular site is amazing – We’re running ads/offers into a few very nichey demographics on that site that generate huge CTR’s (.300-.400; in comparison, the CTR on this campaign is .086 for our best creative) and haven’t beaten this conversion for any other squeeze page.
7. Any lessons which can be derived from your test?
I’m reminded of the parable where an old friend of Henry Ford’s said “Henry, why don’t you ever buy any bolts from me?” to which Mr. Ford replied “Heck Joe, you’ve never asked!” It’s always worth including action verbs like “get”, “watch”, “download”, “enter” etc. in tests.
8. How valuable was Visual Website Optimizer for this test?
It’s so much easier – and more powerful – than Google’s website optimizer. We’re finding it to be an indispensable tool that made me excited about running tests again.
9. Short background information about your business.
The Social Man is in business is to make men awesome. We have a variety of products and coaching packages that cover everything from flirting over text messaging, to how to be sexier, to overcoming social anxiety. I previously ran an enterprise software company, and feel very blessed to be doing something I love that’s employs passionate people and has helped so many men. My business, Turnseven, Inc., manage a few other niche brands in addition to The Social Man.
Posted in A/B Split Testing, Case Studies on December 20th, 2010
Boy, what a year 2010 has been for A/B testing industry and Visual Website Optimizer. We saw tens and hundreds of small/medium sized businesses increase sales and conversions on their websites. Our repository of A/B testing case studies is the proof that A/B testing works! To celebrate the excitement of 2011, I have complied the list of our best A/B testing case studies in 2010 (in order of bestness). Hope they will inspire you to kick-start A/B testing campaign in your organization in 2011.
Removing a signup form actually increased signups by 60%
Original Landing Page (with embedded signup form)
Variation (no signup form): 60% increase in signups!
Who would have thought that removing a signup form from signup page can actually increase conversions by 60%? Vendio, an Alibaba.com company, used Visual Website Optimizer to do an A/B test on their signup landing page. In one version, they removed the signup form altogether and instead wrote about benefits of signing up. The end result was that the version with no signup form eventually saw 60% more signups. Click here to read the full case study (includes screenshots)
Two magical words improve conversion rate by 28%
Control: 14.5% conversion rate
Variation: 28% improvement conversion rate!
Soocial tested multiple variations of their sign up button in order to increase clickthroughs to the registration form. Variations included call-to-actions such as “Sign up for free”, “Free sign up”, etc. The A/B test results revealed that one variation increased conversions dramatically. Click here to read the full case study (includes screenshots)
Human photos double conversion rate
Control (paintings)
Variation (photos): 95% increase in CTR!
In two separate A/B tests, it was found out that human face photos doubled the conversion rate on a website. The result was striking because the tests were conducted by two completely unrelated companies, yet they arrived on the same conclusion. Click here to read the full case study (includes screenshots)
Downloads increased by 85% by testing tour page
Original (control)
Redesigned (variation): 85% increase in downloads!
You Need A Budget makes easy-to-use personal financial management software. They split tested their product tour page by including more screenshots, adding a prominent testimonial and reducing text clutter. The redesigned page increased downloads of the free trial by 85%. Click here to read the full case study (includes screenshots)
Redesign of sales page improves sales by 20%
Original version
Redesigned version: 20% increase in sales!
AquaSoft redesigned their sales page by giving it a modern, clean look. They also added trust building elements such as money-back guarantee, assurance seal, etc. Using Visual Website Optimizer, they effortlessly tested old v/s new sales page. After multiple phases of testing, they successfully increased total sales by 20%. Click here to read the full case study (includes screenshots)
What’s in store for 2011?
Well, we already have a lineup of many interesting split testing case studies. From product pages to eCommerce sites. From effect of video on signups to effect of number of steps in checkout process on sales. We have a case study lined up for all these situations.
To keep updated with our latest A/B testing case studies, subscribe blog updates through email (powered by Feedburner).
Posted in A/B Split Testing, Multivariate Testing on December 10th, 2010
Many people seem to get confused about the terms A/B testing, split testing and multivariate testing. Essentially, A/B testing and split testing are the same concepts but multivariate testing is different. So for all practical concepts, consider A/B Testing = Split Testing. The following table will illustrate difference between A/B testing (or split testing) and Multivariate Testing:
| A/B Testing | Multivariate Testing | |
| What is it? |
In A/B testing you split traffic amongst two or more completely different versions of a webpage (landing page, home page, etc.) The variations of your original page can differ in any manner. You can either just change the headline; or you can even change entire design, layout, offer and what not in the variations. What you change in A/B test is only limited by your creativity. For example, if you are A/B testing on your landing page you may want to create one version with 15% discount, the other with free shipping and one with the same offer but different page design. |
In multivariate test, you identify a few key areas/sections of a page and then create variations for those sections specifically (as opposed to creating variations of whole page in an A/B split test). So for example, in multivariate test you can choose to create different variations for 2 different sections: headline and image. A multivariate testing software will combine all these section specific variations to generate unique versions of page to be tested and then simply split traffic amongst those versions. That is, you will get these variations of the page: Headline1_Image1, Headline1_Image2, Headline2_Image1, Headline2_Image2. |
| How it works? |  |
 |
| Traffic Required |
Relatively Less |
Huge |
| Best used for |
Testing radically different ideas for conversion rate optimization |
Optimizing and refining an existing landing page or homepage without doing significant investment in redesign |
| Example Case Study |
Signups increased by 60% after actually removing the signup form |
I hope that was a good comparison between A/B testing and multivariate testing. If you think I left any point, please let me know. I will update the comparison table accordingly.
Posted in A/B Split Testing, Multivariate Testing on December 6th, 2010
Last year, at the dawn of 2010 I wrote four reasons why 2010 is going to be a year of A/B split testing. Now that 2010 is coming to a close, I thought it will be great to revisit what happened in our tiny little (yet growing) industry of split testing and conversion rate optimization. Undoubtedly, the year saw explosive growth in the industry (and also for our A/B testing tool – Visual Website Optimizer). It was incredibly hard to pick 10 best A/B testing resources (not because there weren’t any, but rather there were simply too many great tools, websites, forums, tutorials, etc.). Anyhow, I managed to compile following is the list of top 10 resources (in no particular order):
WhichTestWon.com
WhichTestWon.com is an interesting website which features a new A/B split test every week. You can vote and discuss about which variation won in the test. It is a fun way to keep learning what usually works for increasing conversion rate on website.
ABTests.com
ABTests.com is an open forum where anyone can share and discuss their A/B test results. ABTests.com and WhichTestWon.com are two great websites when it comes to reading real-life A/B testing case studies. You simply can’t get enough of them!
Ultimate Guide to A/B Testing
The Ultimate Guide To A/B Testing is a guest post I wrote for Smashing Magazine. I had tried to make this post as the definite resource for anyone who wants to get started with A/B testing. Judging from #2 position on Google for ‘A/B testing’ (#1 is Wikipedia), I think I have achieved my goal with that guest post.
#crochat (hashtag for Twitter)
#crochat is a hashtag used by a group of conversion rate optimization enthusiasts to chat on this topic every Thursday. I have seen everyone from experts to vendors to beginners participating in the chat and discussing interesting topics related to A/B testing.
MarketingExperiments.com
MarketingExperiments.com is an organization that does numerous split and multivariate tests every week and have been doing so for years now. They have built an impressive research library which details how they improved website conversions and sales on different kinds of websites: eCommerce, publishers, lead-gen landing pages, etc.
Conversion Conference
Conversion Conference is the first and only conference dedicated to the art of conversion rate optimization. The next edition will be held in San Francisco in March, 2011.
WhichMVT.com
WhichMVT.com is a site where you can compare different multivariate testing tools (including Visual Website Optimizer). You can also read or leave review of different tools. (Just in case you happen to use VWO and love it, please take 5 minutes out to leave a review here).
Online Behavior
Online Behavior is a new magazine by industry experts which features columns on behavioral targeting, web analytics and testing & optimization. All articles are of top-notch quality. If you are interested in conversion rate optimization, subscribing to their feed is a must.
Get Elastic
GetElastic.com exclusively focuses on conversion rate optimization and a/b split testing for eCommerce websites. It’s a must read blog for anyone even remotely related to an eCommerce business.
Visual Website Optimizer Blog
You knew this was coming, didn’t you? 
I am very happy that Visual Website Optimizer Blog has seen explosive growth in readership and reach in last couple of months. This blog is not just about product/feature announcements only. Rather majority of posts relate to A/B testing case studies and conversion rate optimization how-tos. All time hits include: landing page optimization tips, impact of SEO on A/B testing, top 10 eCommerce sites (by conversion rate), etc.
Are there any other A/B testing resources that I missed? Please leave a comment below!
Posted in A/B Split Testing, Case Studies on November 18th, 2010
It is interesting how counter-intuitive A/B testing results can get. One of the best examples of that counter-intuitiveness is a recent A/B split test. This test was done by a Visual Website Optimizer customer Vendio, an Alibaba.com company that specializes in providing free e-Commerce stores to its merchants. They have special landing pages for the free store signup and the objective of this test was to increase signup conversion rate on one of those landing pages.
A/B test details
Their original landing page (control) combined marketing content and registration fields in an attempt to reduce the number of clicks for a successful registration. This is how it looked:
Original Landing Page (with embedded signup form)
Note that it uses the so-called “best-practice” of embedding the signup form in the landing page itself. They had long been using the layout of the original page because as a best practice they presumed that reducing the number of clicks for a registration increased conversion rate. Although the page was performing relatively well, they wanted to make sure the included registration fields weren’t too aggressive or limiting in any manner.
So, the variation that they tested had somewhat similar imagery and content but the page didn’t include any registration fields and had slightly different styling. Clicking on ‘Signup Now’ button simply took the visitor to a page with signup form. In other words, Vendio added an extra step in their conversion funnel. Not a smart move, huh? This is how variation looked like:
Variation (no signup form) – 60% increase in signups!
A/B test results
Guess what? The page without the registration fields performed better – much better – to the tune of a 60% increase in conversions! Here is what Vendio had to say about the results:
Best practices are NOT always true! It’s still hard to believe, but the numbers don’t lie.
So, that’s the biggest lesson here and it is worth repeating: “Best practices are NOT always true”. If you do changes on your website or landing pages without A/B testing them, you are actually flying in the dark. Another lesson here is that it is it is worth testing radically different ideas – which, on the first glance, may appear not-so-smart (like removing signup form from the landing page!).
Testimonial about Visual Website Optimizer
After the successful test, Vendio gave us a great testimonial:
Visual Website Optimizer was extraordinarily valuable. It was far easier to use than other solutions (some of which we couldn’t even get to work properly after implementation). Without VWO we would still be missing out on all those registrations, and we are continuing to see great and other surprising results with VWO.
Posted in A/B Split Testing, Case Studies on October 19th, 2010
Many times A/B testing is not limited to a single conversion goal. In fact, your test variations usually affect many different conversion goals on your site such as free trial signups, paid signups, newsletter subscription, etc. Measuring all these goals for a test is important because a variation may work brilliantly for one goal (e.g. it increases free signups) but may perform worse for your other goals (e.g. decreases your paid signups). If you don’t measure multiple goals for your A/B test, you are essentially flying blind in the dark and may end up making wrong decisions (based on a single conversion goal).
Case Study
A brilliant example of this is a recent A/B test by one of our customers: Guido Jansen (a Magento specialist). He runs Dutchento.org which is the official Dutch community for Magento CMS. The goal of his recent A/B test was to increase subscriptions for Dutchento’s newsletter and RSS subscription. The call-to-action for subscription is located in a box on all pages of the site. See the control version of subscription box below and note that there is no explicit incentive for a visitor to subscribe:
Guido tested a variation of this subscription box which included a title and some benefits in a bullet list. Here is his hypothesis for increasing subscriptions:
People don’t just subscribe to a newsletter or newsfeed for nothing, you should convince them it has added value above just visiting the website. So what I wanted to test is if adding convincing reasons to subscribe would increase the newsletter and newsfeed subscription rate. I measured the impact of the convincing reasons on clicks on both the newsletter and newsfeed links.
Here is how his variation (of subscription box) looks like:
Results
As expected, Guido saw a significant improvement of the newsletter click rate (+190.31%). However, newsfeed click rate decreased (-44.46%) which did surprise him (and us!). He expected that the convincing reasons would affect both positively, but apparently it had a negative effect on newsfeed clicks.
The reason why clicks on newsfeed decreased is not clear but we believe that the benefits in variation were so compelling that visitors chose to get the blog updates via email (where they will be sure to read them) rather than RSS reader (where they may miss them). A great way to get more insight into this would be to randomize the position of newsfeed / newsbrief to eliminate the positional effect of those links.
Testimonial
Guido used Visual Website Optimizer for A/B testing and here is what he had to say:
Visual Website Optimizer was very valuable [for testing]. It’s the easiest A/B and Multivariate testing tool I know. It’s great not to be dependent on the development department to create and run your tests.
Key Lesson
Whatever be the actual reason of decreased clicks on RSS feed, one key lesson is jumps out of the case study: always measure multiple conversion goals in your test. Relying on a single conversion goal hides a lot of valuable information from you. So, make sure you add multiple goals to your next A/B test.
Do you have more examples of using multiple goals in A/B test? What are your thoughts on this case study? Please let us know by leaving a comment below – will be happy to discuss!
Posted in A/B Split Testing, Case Studies on August 2nd, 2010
You Need A Budget (YNAB) is a user-friendly application for personal finance management. The company teaches people how to become awesome at managing their money by following a 4-Rule methodology. And they sell personal budgeting software that follows the methodology. The software is elegant, simple to use, and does just what you need it do – meaning you don’t bog down in unnecessary features that eventually make you want to quit (and pull your hair out).
Using Visual Website Optimizer, they wanted to increase the number of downloads for 7-day free trial of their personal budgeting application. For their first test, they chose to optimize product tour page. For software applications, the tour page is the one which takes up the job of convincing a casual visitor to try out the application. Motivated visitors will anyway download the application so it is the semi-interested visitors who need to be convinced. To optimize the tour page, YNAB chose to do a simple A/B split URL test with the conversion goal as downloading of the free trial.
They designed the variation with following objectives in mind:
- require less clicking by showing all screenshots as nice, large thumbnails
- freshen up the design of the page (having handwriting-type captions)
- strengthen the call to action with a prominently placed testimonial at the very top
As you will see below, the major change was including many more screenshots and hence not requiring visitor to click on different ‘categories’ to see what app can do. You see, semi-interested visitors are more likely to hit the back button than activity hunt for what your app can do. So, including all relevant functionality of the app in form of screenshots on a single page did the trick of convincing them to download the free trial. Here are the screenshots of variations:
Original (control)
Redesigned (variation) – 85% increase in downloads
The variation with more screenshots and lesser categories/links increased downloads by 85%. This result was statistically significant at 97% confidence, so YNAB has implemented it permanently for their tour page. In their own words, here is what they concluded from the test:
We simplified the page presentation and strengthened the call to action with a very convincing and authoritative quote.
Thankfully, like many other customers, they had great praise for Visual Website Optimizer:
Visual Website Optimizer was extremely valuable. We’ve been working with Google’s website optimizer for the past three years and we’re never going back. We were constantly having to jimmy-rig the script code, make sure it was not conflicting with Adwords code, or Google Analytics, etc. It was a big enough nightmare that we avoided testing as much as we could – something that cost us real money over that time obviously.
Are you wondering what is going to be the next page they are going to optimize? Homepage, of course! YNAB will optimize their complete funnel from homepage to product tour to payment. We wish YNAB best of luck for their future tests.
Posted in A/B Split Testing, Multivariate Testing on June 28th, 2010
We recently revamped the introductory video for Visual Website Optimizer by adding a voice-over and updating it by including some new features (e.g. heatmaps).
From start (entering the test page URL) to end (interpreting test results), all it takes to run your first A/B test is just 2 minutes. And you thought testing is a dark art? 
Watch the video below:
Do you think video can be improved? Please leave a comment to let us know what can we add, remove or change to make A/B testing using Visual Website Optimizer even more exciting.
Stay up to date
Follow on Twitter
Email Updates
Subscribe via RSSLatest Posts
Latest Tweets
- RT @unbounce: RT @OliGardner Do long-form sales letters still work? Even for “Professional” businesses and startups? http://t.co/IbjWzW8 ... #
- RT @JeremyReeves: Do long-form sales letters still work? http://t.co/8xo95ei7 via @wingify #
- @juhotunkelo Not yet, but we're working on it. #
- Do long-form sales letters still work? Even for “Professional” businesses and startups? http://t.co/cKI4PdRr #
- 12 Ways to Increase Online Sales http://t.co/X6m2yk8k #
