We just launched a cool new feature in Visual Website Optimizer: email notifications! VWO already has a feature which monitors your test in the background and disables loser variations or starts displaying winning variation to all site traffic automatically. This ensured that the test reduced risk of losing sales and conversions because losing variations get automatically disabled. Now, we have added email notifications to automatic monitoring so that any time a winner or loser variation is found, you get an email like the following:
UWe also did a quick update in conversion goals, so that you can now choose any goal as primary goal (which will be analyzed for crunching email notifications). So, now you can pick the conversion goal (note: you can add multiple conversion goals for each A/B test) you want to monitor, choose to disable losing variations for that goal and simply sit back and relax. You will get updates as soon as losing or winner variations are found!
A customer requested this feature, and we obliged! Visual Website Optimizer reports now come with a date picker to let a user see reports only for a specific date range (of course, in addition to the default aggregated report). See below how this date picker looks like:
Although test reports should always be analyzed in aggregate (to avoid introducing any statistical bias), this feature will come handy if you want to analyze following scenarios:
We are excited about this new feature, and it goes on to show that we are working towards making reports powerful and flexible to allow different kinds of analysis. Hope you too like the new feature!
Note: the date range doesn’t apply to clickmap and heatmap data. They are always shown in aggregate.
To aid better visualization of A/B and multivariate test reports, we recently introduced some new features in Visual Website Optimizer. Even though these are small updates, we are sure they are a step in right direction to provide intuitive understanding of test data.
Test Summary
In Visual Website Optimizer, you can add multiple conversion goals to getter better perspective on performance of variations. For example, we are running an A/B test on homepage where we are testing the word “a/b” v/s “multivariate” v/s “split” in the headline World’s easiest A/B testing tool to see if it makes any impact. And we are tracking five different goals in this test. With the new test summary section in reports, conversion rate can be seen for different variations (on different goals). See a screenshot below (data is fictitious in this case):
The number in parenthesis is the visitor count for the respective variation. So, in a single glimpse you can see how test variations are performing on different conversion goals.
Cumulative Charts
Visual Website Optimizer has nice looking day-wise charts in the test reports which show conversion rate for variations for different days (during the time test is running). Since conversion rate can fluctuate for different days (well, weekends are usually duller than weekdays) the chart usually looks discontinuous and it is hard to find trends in it. Take a look at example of a day-wise chart:
Though it is great at reporting number of visitors, conversions and conversion rate for each day, it isn’t that good at showing trends. So, we decided to supplement it with a chart with cumulative data. That is, a chart where visitors and conversions on a particular day has visitors and conversions added for all previous days. The resultant chart (which we unsurprisingly call Cumulative Chart) produced beautiful trends which you can clearly observe. Have a look below how a cumulative chart looks like for the same data as the above chart:
What’s next for reports?
Data is useless unless you can make it reveal its deep-hidden secrets. So, we want to introduce several new visualizations in test reports. We are currently working on adding funnels into test reports, whereby you can visualize the order in which your visitors complete the goals (for different variations). You will be able to visualize and optimize funnels for different variations for your A/B and multivariate test. We are also working on adding functionality to segment test results by date, so you will be able to pick a period and crunch data only for those dates. (You can even exclude weekends or special promotion days when you know for sure that the data is skewed and biased).
Do you have other ideas for reports and charts that we can build into Visual Website Optimizer? We would love to hear them. Visualization of data is something we absolutely love to discuss! 
With split testing, there is always a fear at back of the mind that the results you observe are not real. This fear becomes especially prominent when you see an underdog variation beating your favorite variation by a huge margin. You may start justifying that the results may be due to chance or you have too less data. So, how do you really make sure that the test results are reliable and can be trusted? Reliability here means that running the test for more duration isn’t going to change the results; whatever results you have now are for real (and not by chance).
So, how do you determine reliability of your A/B test? Hint: you don’t. You let your tool do the work for you. Visual Website Optimizer employs a statistical technique where your conversion events are treated as binomial variables. Above a certain sample size (10-15 visitors), binomial variables can be approximated to a normal distribution. Key point to note is that your conversion rate is a distribution, not a single value. That is, you always get a range (e.g. 31.3% ± 14.8%) for your conversion rate.
As you can see in the image above, a range for conversion rate is provided in the report. For statistics-geeks, this range actually represents 80% of the total area of the normal distribution. One peculiar property of the range (thanks to a concept called standard error) is that initially it is very wide but as more data gets collected, with time, it becomes narrower and narrower. Moreover, with time, the estimate of true conversion rate becomes preciser. For example, if on day 2 of your split test you observe that your conversion rate for a variation is 50% ± 25%, after a week you may observe that it has changed to 40% ± 8% (note that conversion rate has changed and the range has become narrower).
The way to be sure that your results are reliable is to compare conversion rate ranges of different variations and see if there is NO overlap between them. You can visualize this overlap in the chart above. Observe that there is little or no overlap between Control and “free download” variations. So you can be pretty sure that this result is reliable and “free download” indeed works better than the control (which in this case was a simple “download”). This overlap (in distributions) can also be calculated numerically and Visual Website Optimizer calculates it as “Chance to Beat Original” metric. If that value is >95%, you can be pretty confident that the variation will be better than the control.
If you don’t really trust the statistics (by the way, there is no way you shouldn’t), you can still be confident about the test results by employing a neat trick. The idea is to have two identical variations (usually of the control) and see if there is any difference in their conversion rates. Of course, a minor difference will be there (due to randomness) but if you see a large difference between conversion rates of identical variations, you shouldn’t trust the results.
In upcoming posts, I’m going to describe the maths behind split testing, so stay tuned.
