and so should you...
We recently revamped the introductory video for Visual Website Optimizer by adding a voice-over and updating it by including some new features (e.g. heatmaps).
From start (entering the test page URL) to end (interpreting test results), all it takes to run your first A/B test is just 2 minutes. And you thought testing is a dark art? 
Watch the video below:
Do you think video can be improved? Please leave a comment to let us know what can we add, remove or change to make A/B testing using Visual Website Optimizer even more exciting.
Continuing our mission to juice up backend infrastructure of Visual Website Optimizer, today we switched to a Content Delivery Network (Amazon CloudFront) for serving static JavaScript file. Earlier we used Amazon S3 for this purpose (and the reason was this). But since now we have changed the failover mechanism, we can easily move to a CDN for delivering static files.
To cut out the technical details, the change we did today means that the load time due to Visual Website Optimizer will be drastically reduced as now the code that is needed to run a test is downloaded much faster (our benchmarks and early feedback suggest it is at least 2x-10x faster). Following are some FAQs related to this change.
Basic question: what do I need to do if I am currently running tests?
You simply need to update VWO code snippet to the latest one in order to enjoy faster loading tests. The code snippet is available in Add Code to Website section in test reports. By the way, the new URL of the JavaScript file is http://d5phz18u4wuww.cloudfront.net/vis_opt.js
Techie question: why don’t you have a nice URL for JavaScript file? Don’t you know about CNAMEs?
Yes, we are aware of it and desparately wanted to use a URL http://cdn.visualwebsiteoptimizer.com/vis_opt.js however CloudFront doesn’t yet support custom SSL certificates. That is, sites accessing https://cdn.visualwebsiteoptimizer.com/vis_opt.js would have thrown a certificate error which isn’t nice. We will update the code fragment with a nice URL as soon as Amazon starts supporting it.
Offbeat question: what if I want to host static JavaScript file myself?
First, it is not at all needed now as CloudFront will be faster than hosting the file yourself. Second, if you really want to host the file yourself, please go ahead and do it. Though make sure you keep up to the latest version as we regularly do minor (unannounced) bug fixes. However, whenever there is a major change we definitely announce it on blog and newsletter.
Philosophical question: what’s the answer to life?
UPDATE:
To fully convince you how fast Visual Website Optimizer has become, here are some load time metrics from different locations across the world (thanks to WatchMouse).
Load time of previously S3 hosted code snippet (observe that average total load time is about 700-1000 milliseconds).
Load time of new CloudFront hosted code snippet (observe that the average total load time has come down to 50-70 milliseconds – a whopping 10x speed improvement).
Our two main priorities for Visual Website Optimizer are: a) extra load on customers’ website; b) impact on customer’s websites if our servers go down. These issues are super-important to us because unlike most other web applications, our design decisions not only impact our own website but hundreds of other (customer) websites which currently include our JavaScript for doing A/B split tests.
As far as speed is concerned, we did a series of benchmarks recently to determine the bottlenecks. It turned out that the slowest part of the process is 28K (gzipped) JavaScript file download. The file was hosted on Amazon S3 (because of this) and we have now decided to switch to a CDN (Content Delivery Network) for serving the file. Our benchmarks show Amazon CloudFront is about 10 times faster as compared to S3. The reason our JavaScript file is 28K is because it packs JQuery, which is needed for switching page content during A/B test (for technical folks: it is needed for interpreting CSS selectors). We recently experimented with CSSQuery library (which is much lighter) and it brought down the size of JavaScript file from 28K to 8k. However, that was just an experiment. Going forward, we plan to spend more energy in trimming the file size.
As far as failover is concerned, we want to absolutely make sure that impact of our downtime is minimal (idealy nil) on our customers’ websites. We can afford to be down but we cannot afford our customers’ websites to be down because of us. Our uptime has been around 99.8% (see our public uptime reports). We are specifically interested in finding out what is the worst that can possibly happen and try minimizing that. Our research led us to conclude that even for the best configurations in the world (yes, that includes Google, Amazon and all biggies), maximum theoretical downtime (for a few users) will be 30 minutes. You cannot possibly escape this limitation.
In high-availbility systems, failover is typically implemented by using a system called IP takeover using heartbeat. Essentially what this means is that two or more servers share a same public IP address and they monitor each other continuously. In case one goes down, the other servers take up the IP address and begins serving requests. To end user, all this is transparent and failure of one machine doesn’t really affect the service. The one key point here is that for IP takeover mechanism to work, all servers must be in same data-center (and perhaps on the same network switch?). That is, this strategy works if backup servers are physically close to each other.
However, suppose that your data-center suffers a major network and power outage (though uncommon, can still occur). IP takeover isn’t going to help in this case as all servers go down (being in same data-center). To prevent such a scenario, a technique called DNS failover [pdf link] is used. Whenever a user requests your website (www.example.com), browser requests DNS servers to determine which IP address corresponds to that URL. That is, browser contacts DNS servers to resolve www.example.com to an IP address of the main server. In DNS failover, your website is continuously monitored and if it is detected to be down, DNS servers are automatically updated to point www.example.com to a backup server (located in a geographically separated data center). So, all requests for www.example.com which earlier resolved to your main server start resolving to your backup server. This means that if you suffer major outage in our main data center, you can start serve requests from your backup server in a different data center.
However, this is an important caveat. DNS enteries have a Time-To-Live (TTL) values attached to them which tell browser for how long it should cache the response to DNS resolution query. That is, if your DNS entry TTL is 20 minutes, browser will make one DNS lookup for your website and cache the results so that it doesn’t have to lookup results for another 20 minutes again. The downside is that if your server goes down within those 20 minutes, the browser will not see the updated (backup server) IP address for the whole 20 minutes and visitors will see your servers as down. (New visitors will see the backup server as they will make a fresh DNS resolution query).
This can, of course, be solved by having a low TTL value. At Visual Website Optimizer, we are using a TTL value of 2 minutes so that in case of downtime DNS is updated within 2 minutes to reflect backup server availability. That is, within 2 minutes Visual Website Optimizers will appear to be up, even if there is a major outage/downtime. If typical response time is 2 minutes, why do we say theoretical downtime is 30 minutes? That’s because Internet Explorer caches DNS for 30 minutes irrespective of the TTL value. This means that no matter how low is your TTL value, Internet Explorer users who are using your website (at the time it goes down) will see servers remain down for at least 30 minutes. Note that all new users (including IE) will do a fresh DNS query and see your website as up. For more details on IE’s atypical behavior read this PDF document.
If you are Visual Website Optimizer user (or even user of any other service), you now know the maximum theoretical downtime you can expect. For VWO users, because you have JavaScript which contacts our servers, your website will not go down but rather slow down if our servers go down. Note that it is an extreme case as our uptime is already 99.8% which means you may not have to experience any downtime at all. But we believe educating customers on worst-case scenario is beneficial. Also note that you always have an option of self-hosting JavaScript files thereby eliminating all synchronous contact with out servers and avoiding any slowdown at all (in rare cases).
If you are a technical person reading this and think we are misguided in our research, please leave a comment or email us at info@wingify.com. If you have a better solution for reducing downtime impact, we will of course love to be wrong 
Since the launch of the Visual Website Optimizer, we have noticed that it has become a popular testing tool in Europe, especially in the Scandinavian Countries and Western Europe. So in our ambition to become the best A/B split testing tool in the world, we decided it would be a good idea to get our website in different languages. Today we are very proud to announce that our Dutch partnering team has launched VisualWebsiteOptimizer.nl for all our Dutch speaking users!
Why localized website?
Since the Dutch testing audience is a mature and specific market, the Dutch regional team will be maintaining the website in a way that is suitable for their specific target audience. The team will also be featuring Dutch case studies, so if you’ve used the the VWO in the Netherlands or Belgium and have results to share, contact the Dutch team and they will put it on their “Ik hou van A/B testing” blog.
In the end it is our goal to service the needs of all the different target audiences in the best way possible. This is why one of the main goals of the Dutch team (next to evangelizing the VWO of course) will be to gather feedback from the Dutch and Belgium users base. In this way we can improve the tool based upon user suggestion and make it even better! Going forward, if needed, we are going to localize the app itself and/or start training programs in Holland and Belgium.
Keep in touch!
Got nifty feedback, an interesting case study or do you just like to keep in touch with the Dutch team? You can discuss the VWO on the special LinkedIN group (officially endorsed by us) with other online professionals, follow the team on Twitter or simply send them an e-mail!
Happy testing Holland and Belgium!
We had launched clickmaps and heatmaps for A/B tests recently in Visual Website Optimizer. Since then it has become the most popular feature amongst our users. We aren’t surprised by the success of this new feature because it gives a new angle to A/B test results. While you can always get information about conversion rate, bounce rate and other hard metrics from standard reports, visualizing “hotspots” on a page has a charm of its own.
Based on feedback from a user, we rolled out a new minor feature: consolidated clickmaps. Unlike traditional clickmaps, which show results only for a particular page, this new feature will aggregate click activity on ALL your website pages to show which parts get most clicks. This is useful if you have a website that has common template/theme across different pages (e.g. header, footer, sidebar, product box, etc.). Using consolidated clickmaps, you will be able to see statistics for elements common to your website.
This feature came handy when I analyzed visitor activity on my personal blog. All I had to do was to enter a URL pattern (http://www.paraschopra.com/blog*) which instructed VWO to generate a consolidated heatmap for all blog posts (technical detail: VWO combined heatmap data for all pages matching the URL pattern; * matches anything and that’s why the entered URL pattern matched all blog posts).
Here is the consolidated heatmap for all my blog posts:
Contrast this with a heatmap only for a single post:
As you can see, the consolidated heatmap has much more information than a single blog post heatmap. That is the real utility; you get to observe patterns of visitor behavior across your website which you may have otherwise missed or would require you browsing through tens of different heatmaps and then stitching patterns in your head.
Using this feature, you also get to see consolidated clickmaps. Here is a screenshot:
The consolidated clickmap clearly shows that 10% of all clicks on the page occur on Visual Website Optimizer banner on my blog. Some more statistics from the clickmap:
The stats are bit on the lower side but if you observe consolidated heatmap, you will see a lot of click activity on post text. This means that even though visitors aren’t clicking on links, they are still interacting with the post and reading it!
I hope you like the new feature! If you’d like to give it a spin simply login to your account (it is enabled by default for all accounts). Or, if you aren’t an existing user, signup for a 30 day (no-obligations) free trial here.
PS: Technical details. If you are wondering how we are able to generate consolidated heatmap, here is the trick: we don’t store absolute X and Y positions of visitor clicks. Rather we store xpath of the element being clicked. Common elements on the website have same xpath across different pages, hence we are easily able to aggregate that data.
Do you know where visitors are clicking on your website homepage? Do you know which parts of your website are hotspots that attract visitor attention? You can, of course, fire up your web analytics tool and tell me that 20% visitors go to sign up page after visiting homepage but that information doesn’t tell you the whole story.
First, some stats about Visual Website Optimizer homepage: 30% of visitors who come to homepage visit the sign up page too. Of course, this figure is unsurprising because we have a giant Sign up for FREE 30 day trial button right in the middle. So, a lot of visitors must be clicking on it to arrive on the sign up page?
That is what we believed all the while. In Visual Website Optimizer, we had recently released an experimental feature called clickmaps & heatmaps. Clickmaps provide complete click statistics on the web pages for different test variations. Heatmaps help in visualizing hotspots on your webpages where most clicks happen.
Continuing with the story of our homepage, here is a screenshot of heatmap of our homepage:
Can you see the hotspot over tiny Pricing link in the top navigation bar? To know exact statistics, we turn to clickmap of the homepage:
We were completely thrown off our chairs when we realized that 25% of the clicks on the page happen on that tiny link, while the giant Sign up for FREE Trial only gets 5% of the clicks. Moral of the story: the button isn’t playing as important role as we imagined. So, what we now replaced the Sign up button on homepage with Watch a short video button because any way visitors interested in signing up are clicking on Pricing link.
A/B testing and Clickmaps combined
The power of clickmaps & heatmaps increases ten-fold when you combine it with A/B testing. For example, if we replace Pricing link with Free Trial – will it generate a similar distribution of clicks? What if we replace navigation bar from left to the right? What if we change the text of the big button from Sign up for Free Trial to Plans & Pricing? All these are interesting scenarios and we plan to test them one by one.
Recently we completed an A/B test where we tested homepage against a simple page with a text box allowing visitor to do a live demo of the tool on his/her website. While the test results weren’t conclusive, the heatmap provided interesting insights:
The clicks were evenly distributed on different links and elements. Another surprising statistic was that in this variation about 10% clicks happened on Home link in the navigation bar as compared to 2% in our default homepage. This shows that because of the text box a lot of visitors thought that they weren’t on the homepage and tried to go to a page which provides more information.
The qualitative results produced by clickmaps and heatmaps complement quantitative A/B test results such as the one in screenshot below (not related to our homepage test):
The combination of A/B test and clickmaps is a lethal one! Being able to visualize click activity for your different variations gives you tons of ideas and insights beyond what can be captured in hard statistics.
Login to your Visual Website Optimizer account or sign up for a free 30 day trial to let us know if you find the new feature useful. Eagerly waiting for your comments!
PS: You can, of course, produce clickmaps and heatmaps for your website without needing to create an A/B test.
Subscribe blog via RSS
