Visual Website Optimizer

A customer requested this feature, and we obliged! Visual Website Optimizer reports now come with a date picker to let a user see reports only for a specific date range (of course, in addition to the default aggregated report). See below how this date picker looks like:

Although test reports should always be analyzed in aggregate (to avoid introducing any statistical bias), this feature will come handy if you want to analyze following scenarios:

• How do the test variations perform on weekends v/s weekdays (most of the sites get different type of traffic on different days of the week)
• If you have been running a test for long, and one specific day you saw a rush of visitors due to an ad campaign or press mention, you may want to analyze how test variations performed for this “new” kind of visitors
• Most importantly, this feature will be useful to verify your test findings. Suppose you found a winning variation and you disable all other variations from the test. After a few days of running only the winning variation on your test page, you can see if the conversion rate for that period compares to what you saw during the test.

We are excited about this new feature, and it goes on to show that we are working towards making reports powerful and flexible to allow different kinds of analysis. Hope you too like the new feature!

Note: the date range doesn’t apply to clickmap and heatmap data. They are always shown in aggregate.

No Comments »

To aid better visualization of A/B and multivariate test reports, we recently introduced some new features in Visual Website Optimizer. Even though these are small updates, we are sure they are a step in right direction to provide intuitive understanding of test data.

Test Summary

In Visual Website Optimizer, you can add multiple conversion goals to getter better perspective on performance of variations. For example, we are running an A/B test on homepage where we are testing the word “a/b” v/s “multivariate” v/s “split” in the headline World’s easiest A/B testing tool to see if it makes any impact. And we are tracking five different goals in this test. With the new test summary section in reports, conversion rate can be seen for different variations (on different goals). See a screenshot below (data is fictitious in this case):

The number in parenthesis is the visitor count for the respective variation. So, in a single glimpse you can see how test variations are performing on different conversion goals.

Cumulative Charts

Visual Website Optimizer has nice looking day-wise charts in the test reports which show conversion rate for variations for different days (during the time test is running). Since conversion rate can fluctuate for different days (well, weekends are usually duller than weekdays) the chart usually looks discontinuous and it is hard to find trends in it. Take a look at example of a day-wise chart:

Though it is great at reporting number of visitors, conversions and conversion rate for each day, it isn’t that good at showing trends. So, we decided to supplement it with a chart with cumulative data. That is, a chart where visitors and conversions on a particular day has visitors and conversions added for all previous days. The resultant chart (which we unsurprisingly call Cumulative Chart) produced beautiful trends which you can clearly observe. Have a look below how a cumulative chart looks like for the same data as the above chart:

What’s next for reports?

Data is useless unless you can make it reveal its deep-hidden secrets. So, we want to introduce several new visualizations in test reports. We are currently working on adding funnels into test reports, whereby you can visualize the order in which your visitors complete the goals (for different variations). You will be able to visualize and optimize funnels for different variations for your A/B and multivariate test. We are also working on adding functionality to segment test results by date, so you will be able to pick a period and crunch data only for those dates. (You can even exclude weekends or special promotion days when you know for sure that the data is skewed and biased).

Do you have other ideas for reports and charts that we can build into Visual Website Optimizer? We would love to hear them. Visualization of data is something we absolutely love to discuss!

1 Comment »

Graphical visualization of dense information is perhaps the most important component for metrics-heavy apps such as Visual Website Optimizer. Accordingly, intuitive display of A/B test results is a priority for us. In the same spirit, we’re happy to  introduced new test result charts. Take a look at the screenshot below and bask in awesomeness:

On the left Y-axis you can see conversion rates for control and different variations. The red series represents average conversion rate. On the right Y-axis you can see total visitors in the test and total conversions. Also, as you can see, hovering on points pops up a box summarizing that day’s performance of the variation. What is not obvious in the screenshot is that by clicking on series legends, you can switch them on and off. Plus, hovering on series legend highlights the respective series and dims all other series. How cool?

These shiny new charts are powered by an open-source library called Flot. This library is absolutely fantastic and very, very powerful. In fact, it is so powerful that we were able to implement new charts in a single day. Special thanks to @NilsFredrik of Maksimer for giving inputs for the new charts and for insisting that we should use Flot for this purpose. We owe new charts to you, Nils-Fredrik

Do give new charts a spin and let me know your feedback. If you have any ideas on how we can  improve visualization of A/B or multivariate test results, I will be very happy to hear them out. Simply leave a comment on this post and let us start a conversation.

So, do you like new charts?

3 Comments »