To aid better visualization of A/B and multivariate test reports, we recently introduced some new features in Visual Website Optimizer. Even though these are small updates, we are sure they are a step in right direction to provide intuitive understanding of test data.
Test Summary
In Visual Website Optimizer, you can add multiple conversion goals to getter better perspective on performance of variations. For example, we are running an A/B test on homepage where we are testing the word “a/b” v/s “multivariate” v/s “split” in the headline World’s easiest A/B testing tool to see if it makes any impact. And we are tracking five different goals in this test. With the new test summary section in reports, conversion rate can be seen for different variations (on different goals). See a screenshot below (data is fictitious in this case):
The number in parenthesis is the visitor count for the respective variation. So, in a single glimpse you can see how test variations are performing on different conversion goals.
Cumulative Charts
Visual Website Optimizer has nice looking day-wise charts in the test reports which show conversion rate for variations for different days (during the time test is running). Since conversion rate can fluctuate for different days (well, weekends are usually duller than weekdays) the chart usually looks discontinuous and it is hard to find trends in it. Take a look at example of a day-wise chart:
Though it is great at reporting number of visitors, conversions and conversion rate for each day, it isn’t that good at showing trends. So, we decided to supplement it with a chart with cumulative data. That is, a chart where visitors and conversions on a particular day has visitors and conversions added for all previous days. The resultant chart (which we unsurprisingly call Cumulative Chart) produced beautiful trends which you can clearly observe. Have a look below how a cumulative chart looks like for the same data as the above chart:
What’s next for reports?
Data is useless unless you can make it reveal its deep-hidden secrets. So, we want to introduce several new visualizations in test reports. We are currently working on adding funnels into test reports, whereby you can visualize the order in which your visitors complete the goals (for different variations). You will be able to visualize and optimize funnels for different variations for your A/B and multivariate test. We are also working on adding functionality to segment test results by date, so you will be able to pick a period and crunch data only for those dates. (You can even exclude weekends or special promotion days when you know for sure that the data is skewed and biased).
Do you have other ideas for reports and charts that we can build into Visual Website Optimizer? We would love to hear them. Visualization of data is something we absolutely love to discuss! 
Revenue from Google AdSense can be increased in two ways: either increasing total traffic on your page to have more ad impressions available OR optimizing click-through-ratio (CTR) on ads. There are a ton of variables that can affect CTR (and thus the performance of AdSense): number of ads on page, location of ad units, unit type, unit size, color scheme, etc. Because of such a large number of variables, optimizing ad revenue becomes a difficult task.
A/B testing different configurations of AdSense using Visual Website Optimizer is a great way to increase revenue. By testing different aspects of your ad units, you get to know which ones make impact on your revenue. However, split testing AdSense presents some unique challenges:
Recently, one of our customers, Nick Taylor from BrightHub setup a split URL test for AdSense and told us how he did it so that others can benefit from his approach. In his own words:
Right – I understand that it’s not possible to track clicks within an IFRAME. So my work around is to create custom adsense channels/tags that only run against specific tests. VWO won’t be able to tell us the adsense click/revenue data however once I log-in to our adsense dashboard and I run a filtered report specific to the adsense channel tag I was running in my test. I can then collect all that data and tie it back to my test (using a report I’m putting together in excel). The key is that you have to setup AdSense tags specific to each test variant and ensure that when that test variant gets served to the user that you are also showing the user the correct adsense channel specific tag. Because I’m doing simple url split testing and I’m not getting crazy with multi-variant tests we’re able to accomplish this fairly easy. It does take a couple extra steps but it does give me the ability to track AdSense data for each VWO test variant.
In fact, Google itself advocates using custom channels for A/B testing. Here is the AdSense support page: What is A/B testing and how do I do it?
Nick provided with a (fake) screenshot of how channel reports look like while doing A/B testing:
As an example, here are three different templates of articles that BrightHub will be testing using Visual Website Optimizer:
If you are a publisher, you can either test only a specific page on your website and then adopt winning layout site-wide. Or, you can generate different layouts from your backend (depending on URL parameter, e.g. ?version=b), you can use Visual Website Optimizer’s site-wide testing feature to do split URL test across many different pages at a time. Then all you have to do is to look at your channel reports to determine the winning layout.
We wish BrightHub success for their A/B testing their AdSense revenue. If you have any ideas or experience of split testing AdSense revenue, please let us know. We are always eager to know (and write about) how people optimize their ad revenue!
ReplaceDirect, a Visual Website Optimizer customer, is a successful Dutch e-commerce site. It’s one of the biggest retailers in the Netherlands in the field of parts and accessories for notebooks and other mobile equipment. The main focus is on energy products like batteries and power supplies, but the product range is quite extensive.
They recently did an A/B test using Visual Website Optimizer where the winning variation reduced cart abandonment by 25%. For e-commerce sites, cart abandonment is one of the most frustrating aspects of business. After all, there’s no company that likes a potential customer to abandon the purchase after spending tons of time on site research and selecting the right products. Fortunately, cart abandonment can be fixed using A/B testing and this case study demonstrates how ReplaceDirect did it.
A recent study by Forrester found out that shipping costs rank as the number one reason for shopping cart abandonment. Combined with other reasons found out in the research, it can be concluded that visitors abandon their shopping cart because they are unsure about additional costs or surprised to find out (only after entering the payment process) that they are actually charged more than they expected. In order to prevent such surprises, ReplaceDirect already tells the customer in an early stage that no shipping costs will be added. So the customer is clearly informed right from the start. This is exactly what ReplaceDirect wanted to test: will it help reduce cart abandonment when a clear order overview is provided during the initial steps of checkout?
Another cart abandonment study by Paypal
They tested their second step of the checkout procedure. It’s the page where customers are asked to fill out their personal information and shipping address. ReplaceDirect made several changes to the page. Probably the most important change was the insertion of an order overview, comprising the product(s), total costs and delivery date. ReplaceDirect also mentions several benefits at this page; these were changed to more relevant benefits specifically matching this step in the order process. Finally the layouts of the page and the form were changed to yield a cleaner look. Several fields, which were not absolutely necessary, were removed. Following are the screenshots:
Control page
Variation page: 25% reduction in cart abandonment
Their new version performed significantly better, reducing cart abandonments by 25%. As a direct result of this reduction, they saw a 12% increase in sales (14% by the time of pushing all new traffic to the new page). ReplaceDirect says they expected the new version to perform better, but an increase that high turned out to be a pleasant surprise. Well, it is always nice to have a 14% increase in sales without releasing any new products or without spending money on ways to get more traffic to the website.
When we asked them if they had any lessons for other e-commerce sites; here is what they had to say:
Try to give the customers exactly the information they need at that particular page or section and leave out the redundant. Always put yourself in the customer’s shoes. This may not be easy so: test it!
ReplaceDirect also had some praise for Visual Website Optimizer:
VWO was very valuable. The ease to set up a test and real-time results made it great to work with.
We are very happy that ReplaceDirect reported these great results by doing an A/B test using our tool.
If you manage an e-commerce site without doing any A/B split testing, you are certainly leaving a lot of money on the table.
In a typical A/B split test, you test whether replacing an image, headline or button with a variation increases conversion rate. Using Visual Website Optimizer, this is fairly simple. You select the element, create a variation and then VWO replaces the selected element with the variation. Simple and sweet.
However, you may also want to test additional content on a website. That is, test a new element that doesn’t exist on your (control) webpage. For example, if you want to test if adding a new widget to the sidebar decreases bounce rate, how are you going to do it? Remember that you cannot simply do a traditional replacement A/B test because widget doesn’t yet exist. You have two options for such a test: either add an empty HTML tag (such as <div></div>), then simply select that empty tag and create a variation. Or, you can create a completely new page with a widget and do a split URL test.
Unfortunately, both of these options require you to modify your website in some way. What if you don’t want to involve IT team? One of our users, Karsten Lund, faced exact same problem and we helped him do this test without touching the website code or creating any new pages (note that the website already has a common VWO tag that allows one to create any number of tests from VWO interface without touching the website code).
First, here is the control page looks like:
Karsten wanted to test if adding a navigation menu on the left would increase downloads and sales. He was also open to removing the right sidebar as the left one will serve that purpose. To do this A/B test, we used Visual Website Optimizer’s Advanced code mode and added a CSS/JS section in the test. Then we created a variation with following JQuery and CSS code:
HTML content for new section
What the code does is this:
Note that all the code above is inserted in the <head> section of the page, that’s why the container element remains hidden and we wait for document to load before adding the left menu and showing the container element again.
Here is how the variation with left menu looks like:
It works perfectly and we are quite happy that we could help Karsten in setting up this test. Benefit of taking this approach was that Karsten did not have to involve SlideShop’s developers for making changes into the website so the turnaround time for getting the test up and running was quick (less than an hour).
Hopefully you like the technique, if you have any other such tips and tricks for Visual Website Optimizer, do let us know.
We’re excited, very excited. Today we launched a new feature that allows Visual Website Optimizer users to track goal A/B test conversions across multiple different domains and sub-domains without doing any extra settings. Due to restrictions posed by the way cookies work, earlier if multiple domains were involved we instructed users to take a hackish approach to track conversions. But now, it just works out-of-the-box. No extra setting, no code fiddling. Finally, cross-domain tracking is a reality.
Why is this significant?
A lot of internet marketers like to track a visitor across different domains. This isn’t for any malignant purpose, but simply because their services involve multiple domains. Consider following scenarios (all of which our users have come across):
While earlier doing such A/B tests was a painful job because cookies on one domain cannot be accessed from another domain, now (using Visual Website Optimizer) it is as simple as it can get. If you create an A/B or multivariate test which has multiple domains, simply select an option (during test creation) which tells VWO to use third-party cookies for tracking visitors and conversions. After that, your test will work as expected (without doing any other changes) even when multiple different domains are involved!
What are third party cookies and how do they work?
Now here is the interesting part: how do we make cross-domain conversion work? The answer is third party cookies. See the following image to understand how it works:
If you enable third-party cookies option in your test, in addition to storing visitor data (variation shown and conversion goals triggered) in cookies belonging to your domain, Visual Website Optimizer will send that data to our servers too. Once the data has been sent, our servers set cookies for the domain dev.visualwebsiteoptimizer.com. If your test involves another domain, next time your page requests test data, our servers send back visitor data too (because we can access visitor cookies set by our servers). In a way, our servers act as proxy between your multiple different domains and hence conversions can be tracked. Third-party cookies don’t tend to work smoothly on IE, Safari and Opera. However, after crazy amount of research and testing, we made it to work on all major browsers. So, this new method of cross-domain tracking should be fairly accurate.
Note that third-party cookie tracking is disabled by default. You need to enable by selecting an option during test creation process.
Feedback?
We believe that cross-domain tracking is an important milestone in the evolution of Visual Website Optimizer. To best of our knowledge, no other A/B or multivariate testing tool provides this functionality out of the box. So, if you happen to do a test which uses this new feature, please let us know your feedback on effectiveness and accuracy.
