A/B Testing | Landing Page Optimization | Multivariate Testing - Visual Website Optimizer

How does Visual Website Optimizer work?

Simple terms:

After creating a test with different variations of a page, you get a small snippet of code which you need to put into your website only once (you can then create unlimited number of tests on different parts of your website without putting code again and again).
When a visitor visits your page, before displaying the page, VWO code determines which test (if at all) is to be run on that page.
Once it is decided that a test is to be run on the current page, the code locates all the sections on the page you had chosen during the designing phase. If it is a split URL test, VWO redirects the visitor to a randomly chosen URL.
If it is a visual test, out of all possible website variations, one is chosen at random and contents of the located sections are replaced by the code according to the chosen combination.
Finally, the page is displayed to the visitor.
If conversion happens, the code sends information to the server that for the current visitor, a conversion goal has happened.

Technical terms:

The JavaScript code snippet you insert on your pages contacts the VWO server whenever a visitor visits your page.
Based on current URL, the server determines if a test is to be run or not.
There are two parts of response that the server sends: a) static JavaScript code for running the test; b) dynamic test-specific data. Both static and dynamic portions of reponse are heavily cached at the server backend, so response time in minimal. With code minification and gzipping, the size of response decreases even further.
The response that comes back from the server determines which elements of DOM need to be automatically replaced by one of the variations (as contained in server response). In case of click, form submission or engagement as a conversion goal, a VWO function binds to click event on the page that checks if a conversion has occurred.
After section replacements and binding of goals, the page is displayed to the visitor.

How much improvement in conversions can you expect?

Improvement in conversions depends on a lot of factors, some of which are:

Elements of website selected for the test impact the expected improvement in conversions. Best pratice is to vary website elements which are most prominent. If you vary some element (such as footer) which is not very significant as far as conversions is concerned, you shouldn't expect a lot of improvements.
Difference in variations also impacts the conversion rates. If variations are drastically different, you can expect quicker and more significant results. While if the difference in variations is small and unnoticeable (e.g. just a small change in font size), you shouldn't expect a lot of improvements.
Number of variations is yet another factor. Larger the number of variations, higher the chances of coming up with a winner. However, having larger number of variations increases the test duration.

Though improvements in conversions varies according to type of website, typically you should be happy with at least >10% improvements in conversions. In some cases, you can be surprised by getting more than 200% improvement in conversions.

Read existing VWO case studies to get a balanced perspective.

What types of conversion goals are supported by VWO?

Click on a link - you specify a URL (website address), a click to which on your test page will be considered as a conversion goal. For example, if you are optimizing for downloads, you want conversion goal to be the download event. In that case, you will specify the download URL (the value contained in href attribute of a tag) as the conversion goal. When a website visitor clicks on the download link, a conversion goal will be registered by VWO.
Visit to a page - it is the URL on which if a visitor arrives, a conversion goal is registered. For example, if you are optimizing the signup process, you can have the URL of the signup completion page as the conversion goal. If the visitor successfully completes the signup process, he arrives on that URL when a conversion goal is registered by VWO.
Form submission - the conversion goal triggers if the specified form is submitted by the visitor.
Engagement - a conversion goal is triggered if the visitor engages on the page (that is clicks on any link on the page). It is effectively a way to optimize website for bounce rate.
Custom Conversion Goal - this option is for technical users who want to have flexibility of specifying a conversion goal anywhere on the website. You get a JavaScript code which you can insert anywhere on the website. A call to that JavaScript code will trigger the conversion goal. You may want to use the advanced option, if you find the above two types of conversion goals limiting for your use-case.
Revenue Goal - this option enabled you to track revenue metrics like total sales, revenue per visitor, average order value, etc.

Miscellaneous Information

Full factorial approach to multivariate testing
Uses cookies to track visitors
Visitors see same variation even on repeat visits
Conversions can be triggered even on repeat visits
The same test can run on multiple pages on the website
Multiple tests can be run on the same page
The test results are realtime; there is no lag
Read about statistical concepts on which Visual Website Optimizer is based in this blog post

Where is Visual Website Optimizer hosted? How reliable is it?

Visual Website Optimizer is hosted with SoftLayer, one of the most reliable hosting providers out there. Historically, we have seen 99.99% uptime. The static JavaScript is delivered via Amazon Cloudfront and in case of an extremely unlikely event of downtime, our asynchronus JavaScript code is designed to gracefully degrade. Our innovative approach ensures your website doesn't slow down while the Visual Website Optimizer snippet is being loaded.

Visual Website Optimizer has been used on websites of leading companies, banks and government institutes without any hitches.

How secure is Visual Website Optimizer?

We take security of your data very seriously. We support HTTPS and never store your password in plain-text (hash them with a secret salt, for you techies out there) and (honestly!) do all essential security stuff that is supposed to be done. Rest assured that VWO is safe and secure. If you have any specific questions regarding security of VWO, shoot us an email. Visual Website Optimizer also has many enterprise security features as well.

How scalable is Visual Website Optimizer?

We are currently handling billions of requests per month and can easily accomodate millions of additional requests. We can have special provisions for high traffic testing. Make sure you contact us before testing >100,000 visitors per day. Otherwise, signup directly.

How much extra load does VWO create on a website?

Our scripts are (g)zipped to reduce in size and heavily cached at the server-side (hello Memcache and APC) and client-side (browser caching of Javascript). Our servers are pretty fast too (thanks SoftLayer). So additional load is minimal (around 50ms on average). In past, we have experience in playing with realtime behavioral targeting, semantic targeting and other advanced analytics techologies which create tremendous amount of load on servers; yet we have been able to manage it without hiccups. To read more about how our code snippet works, click here.

You can check our ping time statistics here.

Will using Visual Website Optimizer negatively affect your SEO efforts?

Good news, no it will not. This is because your test pages are created dynamically when a visitor visits your page. And for displaying the dynamically generated page, a visitor needs to have both JavaScript and cookies enabled. Googlebot and other search engine crawlers do not support Javascript and cookies, so they see your original page. For search engines your page stays the original one. For more details, read this.

What is testing?

In simple terms, testing is a methodology in which you vary different parts of your website to see what produces maximum results (sales, downloads, clicks, page views, etc). No website is optimal, things which may be obvious to you might be completely befuddling to your visitors. Hence it is very important to keep coming up with testing ideas such as varying your signup forms, ad copy, calls to actions, etc. Aim of website testing is to come up with winning variations of existing page which optimize the objectives of your website.

What is a conversion?

Conversion is a term used for whatever goal your website has. Conversion can mean a download off your site, a signup, a purchase, a video play, etc. Conversion rate is the percentage of visitors that come to your website who complete the conversion goal. Conversion rate is one of the most useful metric to measure website success as it is directly linked to revenues.

What is a control?

In a scientific experiment, all results are benchmarked with a condition where no change has been made. Only after comparing against original settings, effect of new settings in an experiment can be observed. These original settings in an experiment is called a control. Similarly, in a test, you existing website design is used as a control and it is seen what additional improvement do new variations bring.

How testing works?

You come up with multiple hypotheses on what you think can increase website conversions. Based on those hypotheses, you create different variations of your website. These variations can pertain to difference in website color scheme, button sizes, value proposition, calls to action and multiple other different paramaters (that you can possibly think). What happens in testing is that your website visitors are randomly allocated to one of the multiple website variations you create. After a significant number visitors have been included in the test, the software analyzes which website variation resulted in most conversions. That website variation is said to be winning variation. You should then implement that winning variation permanently to enjoy sustained increased conversions.

What is the difference between A/B split and Multivariate Testing?

Split A/B Testing - you vary just one element of your website at a time and see which variation results in maximum conversion rate. The element to be varies could be button color, button size, headline, call to action or even completely different webpages (Split URL Testing).
Multivariate Testing - you select multiple elements/sections on your website and create variations for all the chosen sections. On a landing page, for example, you may select call to action, ad copy and product screenshot as sections. Now for each of the section, you come up with different variations. All these variations are combined to create unique combinations of your website. If there are 2 sections, one with 5 variations and the other with 3 variations, there would be 5*3 = 15 combinations. All these combinations are pitted against each other to observe the combination which converts the maximum. During the test, it is then seen which combination results in maximum conversions.

Visual Website Optimizer

About Visual Website Optimizer

Possible Concerns

General Questions