Challenge: Guess which version is the actual winner in this A/B test.

Prize: Out of all the correct entries, one lucky person will win a full year of $249/month Visual Website Optimizer subscription!! Twelve months of our Large Agency plan is total worth $3000 (12*249).

How to enter?: Simply leave a comment on blog and/or tweet about which variation you think increased conversions by 400%. If you tweet, either use the tweet boxes belows or make sure you cc @wingify so that we can track your entry.

When will be the winner chosen?: on September 8th, along with the actual case study, we will announce the winner of this contest.

About this A/B test case study: one variation had a Security badge in the sidebar and in the other there was no such badge. One of the variations saw an increase of 400% in conversion rate (click on coupon code).

So, which version increased conversions by 400%?

– Version A: with Security badge in sidebar –

– Version B: without Secure badge in sidebar –

Remember, the prize for this contest is a full year of $249/mo Visual Website Optimizer subscription total worth $3000! So, enter take part into contest now either by tweeting your choice above or leaving a comment below. (Bonus: if you do both, your entry will be counted twice.)

Related posts:

1. Visual Website Optimizer – first major update
2. Announcing Visual Website Optimizer API
3. Announcing Dutch edition of Visual Website Optimizer

• Original hypothesis of the test
• Test Results
• Discussion about test results
• Screenshots of variations that were tested

You will thank yourself for recording these four pieces of information. Nothing can be more frustrating than trying to figure out what exactly did you test a month ago! Fortunately, with Visual Website Optimizer, this process is mostly automatic. It always retains the test results and the only manual part, of course, is that you need to write down your thoughts in the Test Notes section.

New Feature: automatically generate screenshots of test variations

Earlier you had to depend on preview feature to see what did you test in previous tests. The preview feature will only work perfectly (and show original variations) if your test page didn’t change. But that is seldom true! (In fact, after a successful test, you will change your test page).

Now, the new feature allows you to automatically generate full-page screenshots of variations and then those screenshots are stored on our servers so that you can revisit what you were testing any time in future. In the preview screen, here is how you take the screenshots:

The screenshots will also be automatically generated when you stop the test, so there is no need to manually do it. Thanks to this Visual Website Optimizer’s new feature, now you never have to worry about remembering what were you exactly testing last month!

We hope this new feature helps you in archiving your A/B and multivariate tests. We have been thinking of extending this feature by including tiny thumbnails within VWO reports and also in future these screenshots will be used while generating a PDF report! Your feedback and feature suggestions on screenshot feature will be great.

Related posts:

1. How to increase downloads for your app by 85%? (Hint: include more screenshots!)
2. New feature: email notifications of test results
3. Consolidated clickmaps and heatmaps: a new method for analyzing visitor activity

The integration of the two tools is dead-simple and Unbounce has put together a post announcing why they recommend Visual Website Optimizer for multivariate testing.

Benefits of Integration

The main benefit, of course, is that using Unbounce you can effortlessly create new landing pages. So, if you want to start a new PPC campaign, for example, you won’t have to wait for your IT team to help you with that. Simply fire up Unbounce, create a new landing page, integrate it with Visual Website Optimizer and start optimizing it.

The main benefits of using Visual Website Optimizer with Unbounce is that you would be able to do the following for your landing pages created in Unbounce:

• Multivariate testing
• Advanced A/B testing
• Heatmaps and Clickmaps

Read more about Visual Website Optimizer + Unbounce and watch an instructional video.

Related posts:

1. Video Tutorial: Landing Page Optimization through Multivariate Testing in just 4 minutes
2. Introducing Browse Mode: create an A/B test even behind the login walls of a website
3. How to create an A/B test on WordPress or Drupal site without adding a single line of code

UWe also did a quick update in conversion goals, so that you can now choose any goal as primary goal (which will be analyzed for crunching email notifications). So, now you can pick the conversion goal (note: you can add multiple conversion goals for each A/B test) you want to monitor, choose to disable losing variations for that goal and simply sit back and relax. You will get updates as soon as losing or winner variations are found!

Related posts:

1. New feature: select date range for Visual Website Optimizer reports
2. Yummier than freshly baked cookies: multiple conversion goals for an A/B or multivariate test
3. Visitor Segmentation for A/B and Multivariate Tests

The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.

All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!

Here are different variations that were tested:

With text buttons	With green buttons	With red buttons

The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)

As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.

The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.

In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.

Related posts:

1. Using A/B split testing to reduce bounce rate by 20% for an eCommerce store
2. Left v/s Right Sidebar – which layout works best?
3. How RIPT Apparel increased their sales by 6.3% using Visual Website Optimizer

There is another breed of testing frameworks known as “server-side” A/B testing. Using such frameworks, no modification happens on browser level. In fact, when the browser requests your page, a randomly picked version of your test page is sent from your server. While this means you need to involve your developers for implementing changes in your website code, it does give you unlimited flexibility of testing.

The server-side A/B testing framework for PHP that we officially support is (unsurprisingly) called phpA/B . It’s free to use (for personal websites). If you know PHP, see the documentation for this library. Just as VWO appeals to marketers because it lets them do A/B testing without making any code changes, we are sure phpA/B will appeal to developers who prefer to dive into code. The library provides integration with Google Analytics out-of-the-box. It also filters out all bots and shows them your control page so that it doesn’t affect your SEO efforts. If you are a Ruby on Rails developer, you may be interested in A/Bingo or Vanity.

Why are we supporting server-side testing frameworks? Aren’t they in competition with us? Not really! Our focus with Visual Website Optimizer is to make A/B and multivariate testing easier by taking technical hassles out of the picture. In fact, ideally, we want our users never to touch any code (be it HTML, JavaScript, CSS or PHP). So, VWO is suited for people who don’t want to rely on developers or IT team for doing even the simplest kind of testing (thereby lengthening the time involved in getting the test up and running).

On the other hand, server-side testing frameworks such as phpA/B are better suited to companies which want to tightly integrate A/B testing in their development process. This is in line with the upcoming “lean-startup” or “test-driven-development” scene.

Different companies, different needs, different A/B testing solutions! We hope you find the best fit for your needs.

Related posts:

1. Host Visual Website Optimizer (JavaScript files) on your server
2. Introducing Browse Mode: create an A/B test even behind the login walls of a website
3. How to visualize A/B test results? Shiny new charts to the rescue

Although test reports should always be analyzed in aggregate (to avoid introducing any statistical bias), this feature will come handy if you want to analyze following scenarios:

• How do the test variations perform on weekends v/s weekdays (most of the sites get different type of traffic on different days of the week)
• If you have been running a test for long, and one specific day you saw a rush of visitors due to an ad campaign or press mention, you may want to analyze how test variations performed for this “new” kind of visitors
• Most importantly, this feature will be useful to verify your test findings. Suppose you found a winning variation and you disable all other variations from the test. After a few days of running only the winning variation on your test page, you can see if the conversion rate for that period compares to what you saw during the test.

We are excited about this new feature, and it goes on to show that we are working towards making reports powerful and flexible to allow different kinds of analysis. Hope you too like the new feature!

Note: the date range doesn’t apply to clickmap and heatmap data. They are always shown in aggregate.

Related posts:

1. New feature: email notifications of test results
2. New reporting features: test summary and cumulative charts
3. Host Visual Website Optimizer (JavaScript files) on your server

Using Visual Website Optimizer, they wanted to increase the number of downloads for 7-day free trial of their personal budgeting application. For their first test, they chose to optimize product tour page. For software applications, the tour page is the one which takes up the job of convincing a casual visitor to try out the application. Motivated visitors will anyway download the application so it is the semi-interested visitors who need to be convinced. To optimize the tour page, YNAB chose to do a simple A/B split URL test with the conversion goal as downloading of the free trial.

They designed the variation with following objectives in mind:

• require less clicking by showing all screenshots as nice, large thumbnails
• freshen up the design of the page (having handwriting-type captions)
• strengthen the call to action with a prominently placed testimonial at the very top

As you will see below, the major change was including many more screenshots and hence not requiring visitor to click on different ‘categories’ to see what app can do. You see, semi-interested visitors are more likely to hit the back button than activity hunt for what your app can do. So, including all relevant functionality of the app in form of screenshots on a single page did the trick of convincing them to download the free trial. Here are the screenshots of variations:

Original (control)

Redesigned (variation) – 85% increase in downloads

The variation with more screenshots and lesser categories/links increased downloads by 85%. This result was statistically significant at 97% confidence, so YNAB has implemented it permanently for their tour page. In their own words, here is what they concluded from the test:

We simplified the page presentation and strengthened the call to action with a very convincing and authoritative quote.

Thankfully, like many other customers, they had great praise for Visual Website Optimizer:

Visual Website Optimizer was extremely valuable. We’ve been working with Google’s website optimizer for the past three years and we’re never going back. We were constantly having to jimmy-rig the script code, make sure it was not conflicting with Adwords code, or Google Analytics, etc. It was a big enough nightmare that we avoided testing as much as we could – something that cost us real money over that time obviously.

Are you wondering what is going to be the next page they are going to optimize? Homepage, of course! YNAB will optimize their complete funnel from homepage to product tour to payment. We wish YNAB best of luck for their future tests.

Related posts:

1. Archive A/B test results by automatically generating screenshots of variations
2. Do trust badges on websites work? Oh yes! 32% increase in conversions
3. Using A/B Split testing to refine your startup’s positioning: 90% increase in conversion rate

Test Summary

In Visual Website Optimizer, you can add multiple conversion goals to getter better perspective on performance of variations. For example, we are running an A/B test on homepage where we are testing the word “a/b” v/s “multivariate” v/s “split” in the headline World’s easiest A/B testing tool to see if it makes any impact. And we are tracking five different goals in this test. With the new test summary section in reports, conversion rate can be seen for different variations (on different goals). See a screenshot below (data is fictitious in this case):

The number in parenthesis is the visitor count for the respective variation. So, in a single glimpse you can see how test variations are performing on different conversion goals.

Cumulative Charts

Visual Website Optimizer has nice looking day-wise charts in the test reports which show conversion rate for variations for different days (during the time test is running). Since conversion rate can fluctuate for different days (well, weekends are usually duller than weekdays) the chart usually looks discontinuous and it is hard to find trends in it. Take a look at example of a day-wise chart:

Though it is great at reporting number of visitors, conversions and conversion rate for each day, it isn’t that good at showing trends. So, we decided to supplement it with a chart with cumulative data. That is, a chart where visitors and conversions on a particular day has visitors and conversions added for all previous days. The resultant chart (which we unsurprisingly call Cumulative Chart) produced beautiful trends which you can clearly observe. Have a look below how a cumulative chart looks like for the same data as the above chart:

What’s next for reports?

Data is useless unless you can make it reveal its deep-hidden secrets. So, we want to introduce several new visualizations in test reports. We are currently working on adding funnels into test reports, whereby you can visualize the order in which your visitors complete the goals (for different variations). You will be able to visualize and optimize funnels for different variations for your A/B and multivariate test. We are also working on adding functionality to segment test results by date, so you will be able to pick a period and crunch data only for those dates. (You can even exclude weekends or special promotion days when you know for sure that the data is skewed and biased).

Do you have other ideas for reports and charts that we can build into Visual Website Optimizer? We would love to hear them. Visualization of data is something we absolutely love to discuss!

Related posts:

1. How to visualize A/B test results? Shiny new charts to the rescue
2. New feature: select date range for Visual Website Optimizer reports
3. New feature: email notifications of test results

A/B testing different configurations of AdSense using Visual Website Optimizer is a great way to increase revenue. By testing different aspects of your ad units, you get to know which ones make impact on your revenue. However, split testing AdSense presents some unique challenges:

• The goal is to not just increase number of clicks, rather an increase in revenue. Google doesn’t provide an API for this, so it is difficult to get this data into a 3rd party app such as Visual Website Optimizer
• Google AdSense Terms of Services has various caveats (such as not modifying it to track clicks, not more than 3 units per page) which make working it a bit difficult

Recently, one of our customers, Nick Taylor from BrightHub setup a split URL test for AdSense and told us how he did it so that others can benefit from his approach. In his own words:

Right – I understand that it’s not possible to track clicks within an IFRAME. So my work around is to create custom adsense channels/tags that only run against specific tests. VWO won’t be able to tell us the adsense click/revenue data however once I log-in to our adsense dashboard and I run a filtered report specific to the adsense channel tag I was running in my test. I can then collect all that data and tie it back to my test (using a report I’m putting together in excel). The key is that you have to setup AdSense tags specific to each test variant and ensure that when that test variant gets served to the user that you are also showing the user the correct adsense channel specific tag. Because I’m doing simple url split testing and I’m not getting crazy with multi-variant tests we’re able to accomplish this fairly easy. It does take a couple extra steps but it does give me the ability to track AdSense data for each VWO test variant.

In fact, Google itself advocates using custom channels for A/B testing. Here is the AdSense support page: What is A/B testing and how do I do it?

Nick provided with a (fake) screenshot of how channel reports look like while doing A/B testing:

As an example, here are three different templates of articles that BrightHub will be testing using Visual Website Optimizer:

• Version A
• Version B
• Version C

If you are a publisher, you can either test only a specific page on your website and then adopt winning layout site-wide. Or, you can generate different layouts from your backend (depending on URL parameter, e.g. ?version=b), you can use Visual Website Optimizer’s site-wide testing feature to do split URL test across many different pages at a time. Then all you have to do is to look at your channel reports to determine the winning layout.

We wish BrightHub success for their A/B testing their AdSense revenue. If you have any ideas or experience of split testing AdSense revenue, please let us know. We are always eager to know (and write about) how people optimize their ad revenue!

Related posts:

1. Easiest way to track conversions across multiple domains (using third party cookies)
2. New reporting features: test summary and cumulative charts
3. A/B test case study: how two magical words increased conversion rate by 28%