Visual Website Optimizer

Is pricing information more important than pricing page design? Does design on pricing page influence conversions at all? These are some of the questions that Basekit.com, world’s most flexible website builder, wanted to answer through their latest A/B test. BaseKit allows you to quickly and simply create, host and manage your own website in 100% W3C-compliant HTML. It’s an awesome tool and we are lucky to have them as one of our customers.

Pricing page A/B testing

This case study is about A/B testing of BaseKit’s pricing page and how a redesign of it increased conversions by 25%. They wanted to increase the number of people who visit ‘Buy Now‘ page after visiting their ‘Plans and Pricing‘ page so that was the conversion goal measured in the test. (For their follow up tests, we recommend Basekit to use Visual Website Optimizer’s revenue tracking feature to make sure that not just conversions but total sales is also measured for different variations. This will get a better idea of performance of variation). The traffic that arrived on pricing page (which was tested) is primarily paid, so it is already focused and targeted. Here is how the pricing page originally looked like:

Original pricing page

Redesigned pricing page

In their own words, the variation had “Bolder, brighter, clearer pricing, nicer design, testimonial, more obvious currency selection.” Here’s how the variation looked like:

Redesigned pricing page (25% increase in conversions)

A/B test results and lessons

The new design reached a 95% statistical significance within 24 hours. It was a consistent 25% improvement for the entire duration of the test. They weren’t surprised, but were reassured to see that our improvements had the desired effect. As a follow up test, they are creating yet another redesign of this new pricing page and then do another A/B test. (Continuous A/B testing is key to increasing conversions. Sometimes you get results, sometimes you don’t but if you do lots of A/B tests, you are bound to see some improvements).

Regarding usefulness of Visual Website Optimizer, here’s what Chris Gilfoy, Head of Search at Basekit had to say:

Hugely valuable. I’ve been a fan of VWO for a long time and it was great to take it to a new brand and show how powerful, yet easy to use it is.

This excellent A/B test shows that not just actual pricing details but design of the pricing page is very important. So, when are you going to test your pricing page? (By the way, we are planning to test our pricing page too! Let us know in the comments if you have any ideas for improvement.)

4 Comments »

Beamax, a Belgium based company, manufactures and distributes projection screens for home cinemas and meeting rooms world-wide. They wanted visitors on homepage to go to a site dedicated to ex-demo and one-off items that are sold directly to consumers. (All other screens are sold indirectly through resellers). They admit that it’s a bit odd to drive away visitors to another site from your main page, but they wanted to clear up some space in the warehouse that was taken up by uncommon items.

A/B testing link colors

So, they decided to do a simple A/B test using Visual Website Optimizer. Just above the product images on homepage, they put a standard link promoting the other website. It said:

Great deals on brand new and ex-demo screens here

To increase clickthroughs on link, they tested a red link (with same text) because they felt it would out-perform the standard blue that they use. Plus, it’s something direct marketers use in “real” mail pieces too. As another variation, they transformed link into something banner-like that they thought would even have more impact. Their hypothesis was that the banner version the sure-fire winner. See screenshots below:

Original page

Page with red link

Page with red banner

What type of link got most clicks?

So, any guesses on which version got maximum clicks: blue, red or banner version? Well, the red link and banner both outperformed the blue link and that wasn’t a surprise. But the eye-opening result was that the red link winning from the banner. The improvement of red link compared to the original blue link was pretty big too: 53.13%. (Note we have some other case studies online which demonstrate how red link outperforms the default link. Here are two examples: PDFProducer case study and GSM.nl case study)

Lessons learned: patience pays

Otto Tromm, CEO of Beamax, stresses the importance of waiting for statistically significant results. He says:

In the early stage of the test, the banner was the big winner. But, over time (when the results got more reliable), the red link outperformed the banner. That taught me not to jump to conclusions.

And it was tempting to declare an early winner, because initial results proved my gut feeling. The test proved me wrong, so it teaches you to stay humble too.

So would I implement a red link vs a banner blindly next time? No, I would test it!

Visual Website Optimizer: how important was it?

Choice of the right tool is certainly very important when are you are doing A/B tests. Beamax chose Visual Website Optimizer for the job (just like thousands of other businesses). Here’s what Otto from Beamax has to say about the tool:

I am not a designer or coder and we use Mod-x and CMS defined templates for nearly all pages. So I neither want to call on experts for every test I do, nor do I want to mess up their work. Visual Website Optimizer made it easy for a non-tech guy to do the tests and keep our designer and programmer focused on their own projects.

With Google’s solution, it was a lot more work to implement tests, which is why I stopped using it. Just couldn’t get it all done myself, which is important when you have an idea and quickly want it implemented.

Hope you liked this case study! If you have any comments or suggestions, we are all ears.

7 Comments »

Red Gate software runs an annual challenge where they buy a small software company for a million dollars. They list a number of requirements that the software company must fulfill. One of the requirements that stood out was about conversion rate. They said:

If you’re selling your product then it must have at least a 10% conversion rate.

This requirement actually made me say “Wow, that’s insane”. Let me elaborate why.

What’s wrong with worrying about conversion rate?

Conversion rate is percentage of visitors who actually bought something on your website. Let’s imagine there are two websites: one sells product X with 5% conversion rate and the other one sells product Y with 10% conversion rate. Now, here is a million dollar question:

Is 10% conversion rate of product Y better than 5% conversion rate of product X?

It’s foolish to even begin answering above question without considering following factors:

• How much do these products sell for? If product X sells for 10 times the price of product Y, clearly average sales price of product X is much better in spite of having 5% conversion rate.
• What is the total traffic on the websites? If product X gets 10,000 visitors a day while product Y only gets 100 visitors, guess which one is minting more money?
• What is the lifetime value of customers? This factor is the biggest reason why comparing conversion rate of different websites/products is a useless exercise. Let’s imagine that product X and Y sell for similar price and get similar amount of traffic. Does that make product Y more valuable (since it has higher conversion rate)? Not necessarily. What if company that makes product X has expert salesman that up-sell and cross-sell tremendously and hence derive much more money from a customer in his lifetime.

In nutshell, conversion rate by itself doesn’t tell much (unless you have extra information like traffic, sales price, lifetime value, traffic mix, etc.) So a website with 1% conversion rate may not necessarily be worse as compared to a website with 10% conversion rate. Conversion rate in isolation is a useless metric.

Increase (or decrease) in conversion rate: that’s what should keep you worried

Conversion rates are not entirely useless. In fact, they are very useful when seen on a temporal scale. In other words:

If your conversion rate is 5% today, aim should be to increase it to 7% (using A/B testing, etc,) or at least not let it fall to 3%.

So, comparing conversion rate over time makes a lot of sense (but for the same website). Unless you have a lot of other information about your competitors, you should NOT obsess over comparing your conversion rate to their conversion rate and whether it is lower/higher. Instead, you should obsess on how you can increase your conversion rate (since that’s one of the easiest things to make your bank balance fatter).

Note: if you go through our library of A/B testing case studies, you will note that we always talk about increase in conversion rate and not conversion rate per se.

5 Comments »

Last year, at the dawn of 2010 I wrote four reasons why 2010 is going to be a year of A/B split testing. Now that 2010 is coming to a close, I thought it will be great to revisit what happened in our tiny little (yet growing) industry of split testing and conversion rate optimization. Undoubtedly, the year saw explosive growth in the industry (and also for our A/B testing tool – Visual Website Optimizer). It was incredibly hard to pick 10 best A/B testing resources (not because there weren’t any, but rather there were simply too many great tools, websites, forums, tutorials, etc.). Anyhow, I managed to compile following is the list of top 10 resources (in no particular order):

WhichTestWon.com

WhichTestWon.com is an interesting website which features a new A/B split test every week. You can vote and discuss about which variation won in the test. It is a fun way to keep learning what usually works for increasing conversion rate on website.

ABTests.com

ABTests.com is an open forum where anyone can share and discuss their A/B test results. ABTests.com and WhichTestWon.com are two great websites when it comes to reading real-life A/B testing case studies. You simply can’t get enough of them!

Ultimate Guide to A/B Testing

The Ultimate Guide To A/B Testing is a guest post I wrote for Smashing Magazine. I had tried to make this post as the definite resource for anyone who wants to get started with A/B testing. Judging from #2 position on Google for ‘A/B testing’ (#1 is Wikipedia), I think I have achieved my goal with that guest post.

#crochat (hashtag for Twitter)

#crochat is a hashtag used by a group of conversion rate optimization enthusiasts to chat on this topic every Thursday. I have seen everyone from experts to vendors to beginners participating in the chat and discussing interesting topics related to A/B testing.

MarketingExperiments.com

MarketingExperiments.com is an organization that does numerous split and multivariate tests every week and have been doing so for years now. They have built an impressive research library which details how they improved website conversions and sales on different kinds of websites: eCommerce, publishers, lead-gen landing pages, etc.

Conversion Conference

Conversion Conference is the first and only conference dedicated to the art of conversion rate optimization. The next edition will be held in San Francisco in March, 2011.

WhichMVT.com

WhichMVT.com is a site where you can compare different multivariate testing tools (including Visual Website Optimizer). You can also read or leave review of different tools. (Just in case you happen to use VWO and love it, please take 5 minutes out to leave a review here).

Online Behavior

Online Behavior is a new magazine by industry experts which features columns on behavioral targeting, web analytics and testing & optimization. All articles are of top-notch quality. If you are interested in conversion rate optimization, subscribing to their feed is a must.

Get Elastic

GetElastic.com exclusively focuses on conversion rate optimization and a/b split testing for eCommerce websites. It’s a must read blog for anyone even remotely related to an eCommerce business.

Visual Website Optimizer Blog

You knew this was coming, didn’t you? I am very happy that Visual Website Optimizer Blog has seen explosive growth in readership and reach in last couple of months. This blog is not just about product/feature announcements only. Rather majority of posts relate to A/B testing case studies and conversion rate optimization how-tos. All time hits include: landing page optimization tips, impact of SEO on A/B testing, top 10 eCommerce sites (by conversion rate), etc.

Are there any other A/B testing resources that I missed? Please leave a comment below!

7 Comments »

I just published a guest post on Smashing Magazine titled Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates.

Essentially, there are five steps to increasing conversion rate:

• Identify a challenge
• Define your test hypothesis
• Decide whether to do A/B testing or multivariate testing
• Run the test and analyze results
• Derive lessons from it

If these steps sound complicated to you, I recommend you to read the extensive tutorial which has numerous examples.

My article explains multivariate testing by means of a case study where I tested following variations on a software download page (notice color and text changes):

Can you guess which variation produced maximum downloads? Well, the end result of this test was that #10 combination (in the screenshot above, one with ‘Download for Free’ in red) had 60% improvement in conversion rate. That’s the power of multivariate testing.

Read the full case study and tutorial: Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates. I hope you like it!

No Comments »

When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.

Background

What was tested

Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.

Results

As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.

Control: 14.5% conversion rate

Variation: 18.6% conversion rate

The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.

Why “It’s Free” Worked

When we asked Soocial why they thought the winning variation worked, this is what they had to say:

Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.

So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:

To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.

For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.

How valuable was Visual Website Optimizer for the A/B Test?

Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:

Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.

Key Takeaways

Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!

42 Comments »

In the last two days, we have released a bunch of small, unrelated new features in Visual Website Optimizer. Here they are:

• Exclusion of IPs: this was a much-requested feature to be able to exclude certain office IPs from getting tracked during tests. This wasn’t previously possible and hence caused some bias in the test results as users’ own hits and conversions got counted too. But now you can exclude certain IPs from getting counted in the results. Simply enter the list of IPs to exclude in Account Management section.
• New How-to page: since, feature-wise, VWO has expanded a lot, we thought it will be a great idea to list all common ‘How do I do X’ questions in one page. This is available under Help tab in VWO.
• www and non-www URLs issue: earlier there used to be a lot of confusion amongst new users who entered their URLs as http://domain.com but their actual site was http://www.domain.com/. VWO used to treat both these URLs as different and hence tests did not run properly in such cases. The user had to use a URL pattern (http://*domain.com) to make it work for both cases. Now, VWO simply treats both URLs as same so tests will run on all URLs without any hassle. (We already do a lot of URL trickery. Read about it here.)
• VWO code without JQuery: many of our users have JQuery already present on their websites, so we now provide an option to use (much-lighter in size) VWO code that does not contain JQuery. If you too have JQuery in your website, you can use the new code provided in your test reports.

Hope you liked the new features! As always, if you have any feedback, leave a comment here or email us at info@wingify.com

No Comments »

Most of the times you create and run A/B tests on all your website visitors. But there may be cases when you would like to test your ideas only for a particular segment for visitors. For example, you may want to test if a stripped down layout of website works better for returning visitors. Or you may want to test if a Twitter-specific banner gets more clicks from visitors arriving from Twitter.

New feature: Segmentation

We are proud to announce that we have launched Segmentation feature in Visual Website Optimizer. Using this feature you can choose to run the test only for the visitors who qualify your segmentation conditions. Have a look at the following screenshot:

You can choose and combine any of the following conditions available for segmentation:

• Current URL
• Referring URL
• Visitor Type: New or Returning
• Search Keywords
• Cookie Value
• Referral Type: Direct, Organic or Referrer
• Query parameter in URL (e.g. campaign_id, page_id, etc.)

Through the seven conditions above, we have covered most of the segmentation scenarios. Is there any other condition that you think could be useful for segmentation?

Behavioral Targeting and Personalization Possibilities

This is the most interesting part of segmentation. Thanks to flexibility of running multiple tests on the same page, you can personalize landing pages according to visitor segment. For example, if you have a generic landing page for all your marketing campaigns, using segmentation you can show campaign specific headlines and images to different types of visitors. In other words, thanks to segmentation feature, you can target your visitors with segment specific landing page and test it against control if it really drives up conversions. (You can also do neat stuff like inserting search keywords into variation or personalizing page according to city or country. See Advanced code mode for details).

We are *VERY* excited about segmentation feature as it allows power users to run targeted tests and personalize experience for visitors using a super-easy WYSIWYG editor. If you have any ideas or suggestions, please leave a comment below.

Do you like the segmentation feature? Would you actually see a use-case for it on your site?

14 Comments »

Quick, imagine that you own a small local business, say an automotive repair shop, and you are looking forward to using Internet for getting more business. While researching for your options for tools or services, which of the following pitches is going to appeal you most?

• Businesses grow faster online!
• Online advertising that works!
• Get found faster!
• Create a webpage for your business

Having difficulty making your mind? You are not alone. CityCliq provides low-cost, search engine-optimized webpages for small and large businesses. As they provide a suite of tools to businesses for creating and managing their online presence, getting the positioning right is critical. They recently used Visual Website Optimizer to A/B test which product positioning works best. The conversion goal for this test was a click on their pricing plan. Implementation wise, this simple test was quick to setup (took them just 15 minutes to do so); major effort was invested in coming up with good candidates for positioning. They ran the test for 2 weeks on their homepage and were really happy with the results.

(Click to Expand)

Here is how different positioning statements fare against each other:

About the results, here is what CityCliq had to say:

“Create a webpage for your business was our last test tagline and eventually showed a 98% chance of beating the original. In retrospect, its success makes sense, as it’s perhaps the purest, most direct representation of our product. It didn’t surprise me, as user behavior seems to vary wildly on the web. Plus, there hasn’t been much research done on the behavior of our target audience, so these results were more informative than surprising. “

What they mean is that this A/B test yielded more than just an increase in conversion rate. The test informed them about the psychology of their target customer. This is a perfect example of how A/B split testing can be used to peek into the minds of customers to see misalignment of what they want and what they think you offer. Create a webpage for your business as the winning variation tells that CityCliq’s prospective customers are already aware of benefits of being online and in fact are looking for a tool make that process simpler. All other variations tried to pitch the advantages of being online, which clearly isn’t that appealing to the customers.

Another point to note is that all but the winning variation had an exclamation mark. It would be interesting if they follow up the test with two versions of the winning variation – one with exclamation mark and one without. After A/B testing was completed, CityCliq implemented the winning variation permanently on homepage (as you can see on screenshot) and subsequently saw a rise in user actions. Inspired by their first successful test, they hope to setup a lot of A/B split tests in near future.

12 Comments »