Visual Website Optimizer

Today we’ve released a change in Visual Website Optimizer code snippet that will make the process of integrating VWO to your website a lot easier! As you will see, the change is a significant improvement over the previous code snippet and we recommend you to update it on your website as soon as possible to enjoy all its benefits (of course, it is backward compatible; So your existing VWO code will also work — but the new one is much faster).

The new code is cleaner and much easier to implement

The old snippet required you to insert code at two places: one in header and one in footer. The new code requires only one small snippet to be inserted into the website header. Yes, that’s it! No more you are required to add separate code snippets in header and footer.

Now, all the code required to run VWO, create and run unlimited number of A/B and multivariate tests, measure conversion rates, track heatmaps and clickmaps, etc. is contained in one snippet at a single location. This is what we call as single-tag integration! See, the screenshot below on which code snippet you need to add to integrate VWO with your website:

The new code is faster

The old code downloaded the static JavaScript file (required to run VWO) on all pages irrespective of whether a test is running on the page or not. It meant that the static file was downloaded for all your site visitors even if you are not running any test or test is only running on certain parts of your website.

The new code downloads the static JavaScript file only if and when it is needed. That is, the file is downloaded if a test is to be run on a particular page. Otherwise, it is simply skipped. What this means is a much faster experience for visitors to your website who aren’t included in the test.

We strongly recommend you to update to the new code! You can find the updated code in your test reports (under section Add code to website section)

3 Comments »

We just launched a cool new feature in Visual Website Optimizer: email notifications! VWO already has a feature which monitors your test in the background and disables loser variations or starts displaying winning variation to all site traffic automatically. This ensured that the test reduced risk of losing sales and conversions because losing variations get automatically disabled. Now, we have added email notifications to automatic monitoring so that any time a winner or loser variation is found, you get an email like the following:

UWe also did a quick update in conversion goals, so that you can now choose any goal as primary goal (which will be analyzed for crunching email notifications). So, now you can pick the conversion goal (note: you can add multiple conversion goals for each A/B test) you want to monitor, choose to disable losing variations for that goal and simply sit back and relax. You will get updates as soon as losing or winner variations are found!

1 Comment »

It has been criticized, but it is always guaranteed to work. What is that we are talking about? Yes, you guessed it right: red buttons! No matter how many people consider such A/B tests as a trivial exercise, every now and then they have been demonstrated to increase conversions.

The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.

All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!

Here are different variations that were tested:

With text buttons	With green buttons	With red buttons

The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)

As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.

The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.

In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.

3 Comments »

A customer requested this feature, and we obliged! Visual Website Optimizer reports now come with a date picker to let a user see reports only for a specific date range (of course, in addition to the default aggregated report). See below how this date picker looks like:

Although test reports should always be analyzed in aggregate (to avoid introducing any statistical bias), this feature will come handy if you want to analyze following scenarios:

• How do the test variations perform on weekends v/s weekdays (most of the sites get different type of traffic on different days of the week)
• If you have been running a test for long, and one specific day you saw a rush of visitors due to an ad campaign or press mention, you may want to analyze how test variations performed for this “new” kind of visitors
• Most importantly, this feature will be useful to verify your test findings. Suppose you found a winning variation and you disable all other variations from the test. After a few days of running only the winning variation on your test page, you can see if the conversion rate for that period compares to what you saw during the test.

We are excited about this new feature, and it goes on to show that we are working towards making reports powerful and flexible to allow different kinds of analysis. Hope you too like the new feature!

Note: the date range doesn’t apply to clickmap and heatmap data. They are always shown in aggregate.

No Comments »

You Need A Budget (YNAB) is a user-friendly application for personal finance management. The company teaches people how to become awesome at managing their money by following a 4-Rule methodology. And they sell personal budgeting software that follows the methodology. The software is elegant, simple to use, and does just what you need it do – meaning you don’t bog down in unnecessary features that eventually make you want to quit (and pull your hair out).

Using Visual Website Optimizer, they wanted to increase the number of downloads for 7-day free trial of their personal budgeting application. For their first test, they chose to optimize product tour page. For software applications, the tour page is the one which takes up the job of convincing a casual visitor to try out the application. Motivated visitors will anyway download the application so it is the semi-interested visitors who need to be convinced. To optimize the tour page, YNAB chose to do a simple A/B split URL test with the conversion goal as downloading of the free trial.

They designed the variation with following objectives in mind:

• require less clicking by showing all screenshots as nice, large thumbnails
• freshen up the design of the page (having handwriting-type captions)
• strengthen the call to action with a prominently placed testimonial at the very top

As you will see below, the major change was including many more screenshots and hence not requiring visitor to click on different ‘categories’ to see what app can do. You see, semi-interested visitors are more likely to hit the back button than activity hunt for what your app can do. So, including all relevant functionality of the app in form of screenshots on a single page did the trick of convincing them to download the free trial. Here are the screenshots of variations:

Original (control)

Redesigned (variation) – 85% increase in downloads

The variation with more screenshots and lesser categories/links increased downloads by 85%. This result was statistically significant at 97% confidence, so YNAB has implemented it permanently for their tour page. In their own words, here is what they concluded from the test:

We simplified the page presentation and strengthened the call to action with a very convincing and authoritative quote.

Thankfully, like many other customers, they had great praise for Visual Website Optimizer:

Visual Website Optimizer was extremely valuable. We’ve been working with Google’s website optimizer for the past three years and we’re never going back. We were constantly having to jimmy-rig the script code, make sure it was not conflicting with Adwords code, or Google Analytics, etc. It was a big enough nightmare that we avoided testing as much as we could – something that cost us real money over that time obviously.

Are you wondering what is going to be the next page they are going to optimize? Homepage, of course! YNAB will optimize their complete funnel from homepage to product tour to payment. We wish YNAB best of luck for their future tests.

2 Comments »

To aid better visualization of A/B and multivariate test reports, we recently introduced some new features in Visual Website Optimizer. Even though these are small updates, we are sure they are a step in right direction to provide intuitive understanding of test data.

Test Summary

In Visual Website Optimizer, you can add multiple conversion goals to getter better perspective on performance of variations. For example, we are running an A/B test on homepage where we are testing the word “a/b” v/s “multivariate” v/s “split” in the headline World’s easiest A/B testing tool to see if it makes any impact. And we are tracking five different goals in this test. With the new test summary section in reports, conversion rate can be seen for different variations (on different goals). See a screenshot below (data is fictitious in this case):

The number in parenthesis is the visitor count for the respective variation. So, in a single glimpse you can see how test variations are performing on different conversion goals.

Cumulative Charts

Visual Website Optimizer has nice looking day-wise charts in the test reports which show conversion rate for variations for different days (during the time test is running). Since conversion rate can fluctuate for different days (well, weekends are usually duller than weekdays) the chart usually looks discontinuous and it is hard to find trends in it. Take a look at example of a day-wise chart:

Though it is great at reporting number of visitors, conversions and conversion rate for each day, it isn’t that good at showing trends. So, we decided to supplement it with a chart with cumulative data. That is, a chart where visitors and conversions on a particular day has visitors and conversions added for all previous days. The resultant chart (which we unsurprisingly call Cumulative Chart) produced beautiful trends which you can clearly observe. Have a look below how a cumulative chart looks like for the same data as the above chart:

What’s next for reports?

Data is useless unless you can make it reveal its deep-hidden secrets. So, we want to introduce several new visualizations in test reports. We are currently working on adding funnels into test reports, whereby you can visualize the order in which your visitors complete the goals (for different variations). You will be able to visualize and optimize funnels for different variations for your A/B and multivariate test. We are also working on adding functionality to segment test results by date, so you will be able to pick a period and crunch data only for those dates. (You can even exclude weekends or special promotion days when you know for sure that the data is skewed and biased).

Do you have other ideas for reports and charts that we can build into Visual Website Optimizer? We would love to hear them. Visualization of data is something we absolutely love to discuss!

1 Comment »

Revenue from Google AdSense can be increased in two ways: either increasing total traffic on your page to have more ad impressions available OR optimizing click-through-ratio (CTR) on ads. There are a ton of variables that can affect CTR (and thus the performance of AdSense): number of ads on page, location of ad units, unit type, unit size, color scheme, etc. Because of such a large number of variables, optimizing ad revenue becomes a difficult task.

A/B testing different configurations of AdSense using Visual Website Optimizer is a great way to increase revenue. By testing different aspects of your ad units, you get to know which ones make impact on your revenue. However, split testing AdSense presents some unique challenges:

• The goal is to not just increase number of clicks, rather an increase in revenue. Google doesn’t provide an API for this, so it is difficult to get this data into a 3rd party app such as Visual Website Optimizer
• Google AdSense Terms of Services has various caveats (such as not modifying it to track clicks, not more than 3 units per page) which make working it a bit difficult

Recently, one of our customers, Nick Taylor from BrightHub setup a split URL test for AdSense and told us how he did it so that others can benefit from his approach. In his own words:

Right – I understand that it’s not possible to track clicks within an IFRAME. So my work around is to create custom adsense channels/tags that only run against specific tests. VWO won’t be able to tell us the adsense click/revenue data however once I log-in to our adsense dashboard and I run a filtered report specific to the adsense channel tag I was running in my test. I can then collect all that data and tie it back to my test (using a report I’m putting together in excel). The key is that you have to setup AdSense tags specific to each test variant and ensure that when that test variant gets served to the user that you are also showing the user the correct adsense channel specific tag. Because I’m doing simple url split testing and I’m not getting crazy with multi-variant tests we’re able to accomplish this fairly easy. It does take a couple extra steps but it does give me the ability to track AdSense data for each VWO test variant.

In fact, Google itself advocates using custom channels for A/B testing. Here is the AdSense support page: What is A/B testing and how do I do it?

Nick provided with a (fake) screenshot of how channel reports look like while doing A/B testing:

As an example, here are three different templates of articles that BrightHub will be testing using Visual Website Optimizer:

• Version A
• Version B
• Version C

If you are a publisher, you can either test only a specific page on your website and then adopt winning layout site-wide. Or, you can generate different layouts from your backend (depending on URL parameter, e.g. ?version=b), you can use Visual Website Optimizer’s site-wide testing feature to do split URL test across many different pages at a time. Then all you have to do is to look at your channel reports to determine the winning layout.

We wish BrightHub success for their A/B testing their AdSense revenue. If you have any ideas or experience of split testing AdSense revenue, please let us know. We are always eager to know (and write about) how people optimize their ad revenue!

No Comments »

ReplaceDirect, a Visual Website Optimizer customer, is a successful Dutch e-commerce site. It’s one of the biggest retailers in the Netherlands in the field of parts and accessories for notebooks and other mobile equipment. The main focus is on energy products like batteries and power supplies, but the product range is quite extensive.

They recently did an A/B test using Visual Website Optimizer where the winning variation reduced cart abandonment by 25%. For e-commerce sites, cart abandonment is one of the most frustrating aspects of business. After all, there’s no company that likes a potential customer to abandon the purchase after spending tons of time on site research and selecting the right products. Fortunately, cart abandonment can be fixed using A/B testing and this case study demonstrates how ReplaceDirect did it.

A recent study by Forrester found out that shipping costs rank as the number one reason for shopping cart abandonment. Combined with other reasons found out in the research, it can be concluded that visitors abandon their shopping cart because they are unsure about additional costs or surprised to find out (only after entering the payment process) that they are actually charged more than they expected. In order to prevent such surprises, ReplaceDirect already tells the customer in an early stage that no shipping costs will be added. So the customer is clearly informed right from the start. This is exactly what ReplaceDirect wanted to test: will it help reduce cart abandonment when a clear order overview is provided during the initial steps of checkout?

Another cart abandonment study by Paypal

They tested their second step of the checkout procedure. It’s the page where customers are asked to fill out their personal information and shipping address. ReplaceDirect made several changes to the page. Probably the most important change was the insertion of an order overview, comprising the product(s), total costs and delivery date. ReplaceDirect also mentions several benefits at this page; these were changed to more relevant benefits specifically matching this step in the order process. Finally the layouts of the page and the form were changed to yield a cleaner look. Several fields, which were not absolutely necessary, were removed. Following are the screenshots:

Control page

Variation page: 25% reduction in cart abandonment

Their new version performed significantly better, reducing cart abandonments by 25%. As a direct result of this reduction, they saw a 12% increase in sales (14% by the time of pushing all new traffic to the new page). ReplaceDirect says they expected the new version to perform better, but an increase that high turned out to be a pleasant surprise. Well, it is always nice to have a 14% increase in sales without releasing any new products or without spending money on ways to get more traffic to the website.

When we asked them if they had any lessons for other e-commerce sites; here is what they had to say:

Try to give the customers exactly the information they need at that particular page or section and leave out the redundant. Always put yourself in the customer’s shoes. This may not be easy so: test it!

ReplaceDirect also had some praise for Visual Website Optimizer:

VWO was very valuable. The ease to set up a test and real-time results made it great to work with.

We are very happy that ReplaceDirect reported these great results by doing an A/B test using our tool.
If you manage an e-commerce site without doing any A/B split testing, you are certainly leaving a lot of money on the table.

No Comments »

We’re excited, very excited. Today we launched a new feature that allows Visual Website Optimizer users to track goal A/B test conversions across multiple different domains and sub-domains without doing any extra settings. Due to restrictions posed by the way cookies work, earlier if multiple domains were involved we instructed users to take a hackish approach to track conversions. But now, it just works out-of-the-box. No extra setting, no code fiddling. Finally, cross-domain tracking is a reality.

Why is this significant?

A lot of internet marketers like to track a visitor across different domains. This isn’t for any malignant purpose, but simply because their services involve multiple domains. Consider following scenarios (all of which our users have come across):

• Doing a site-wide A/B test where www.example.com is the main website and www.example.org is the variation
• Involvement of third-party payment processor. For example, if your site is located at www.mystore.com and your conversion goal is visit to a page www.shoppingcart.com/purchase.html
• Suppose you have an international website and you want to do a consistent test across www.example.com and www.example.nl
• You redirect visitors from multiple different lead gen pages to a main product website

While earlier doing such A/B tests was a painful job because cookies on one domain cannot be accessed from another domain, now (using Visual Website Optimizer) it is as simple as it can get. If you create an A/B or multivariate test which has multiple domains, simply select an option (during test creation) which tells VWO to use third-party cookies for tracking visitors and conversions. After that, your test will work as expected (without doing any other changes) even when multiple different domains are involved!

What are third party cookies and how do they work?

Now here is the interesting part: how do we make cross-domain conversion work? The answer is third party cookies. See the following image to understand how it works:

If you enable third-party cookies option in your test, in addition to storing visitor data (variation shown and conversion goals triggered) in cookies belonging to your domain, Visual Website Optimizer will send that data to our servers too. Once the data has been sent, our servers set cookies for the domain dev.visualwebsiteoptimizer.com. If your test involves another domain, next time your page requests test data, our servers send back visitor data too (because we can access visitor cookies set by our servers). In a way, our servers act as proxy between your multiple different domains and hence conversions can be tracked. Third-party cookies don’t tend to work smoothly on IE, Safari and Opera. However, after crazy amount of research and testing, we made it to work on all major browsers. So, this new method of cross-domain tracking should be fairly accurate.

Note that third-party cookie tracking is disabled by default. You need to enable by selecting an option during test creation process.

Feedback?

We believe that cross-domain tracking is an important milestone in the evolution of Visual Website Optimizer. To best of our knowledge, no other A/B or multivariate testing tool provides this functionality out of the box. So, if you happen to do a test which uses this new feature, please let us know your feedback on effectiveness and accuracy.

3 Comments »

We recently revamped the introductory video for Visual Website Optimizer by adding a voice-over and updating it by including some new features (e.g. heatmaps).

From start (entering the test page URL) to end (interpreting test results), all it takes to run your first A/B test is just 2 minutes. And you thought testing is a dark art? Watch the video below:

Do you think video can be improved? Please leave a comment to let us know what can we add, remove or change to make A/B testing using Visual Website Optimizer even more exciting.

No Comments »