A multivariate or A/B test doesn’t end when you stop it. It is important to retain the test if you decide to revisit it later. A good archive of past tests is a powerful information repository; an organization should know what it had previously tested and what the results were. The knowledge derived from previous tests can be of great use in deciding what to test in future. For any A/B or multivariate test, you should aim to record the following before completely stopping it:
You will thank yourself for recording these four pieces of information. Nothing can be more frustrating than trying to figure out what exactly did you test a month ago! Fortunately, with Visual Website Optimizer, this process is mostly automatic. It always retains the test results and the only manual part, of course, is that you need to write down your thoughts in the Test Notes section.
New Feature: automatically generate screenshots of test variations
Earlier you had to depend on preview feature to see what did you test in previous tests. The preview feature will only work perfectly (and show original variations) if your test page didn’t change. But that is seldom true! (In fact, after a successful test, you will change your test page).
Now, the new feature allows you to automatically generate full-page screenshots of variations and then those screenshots are stored on our servers so that you can revisit what you were testing any time in future. In the preview screen, here is how you take the screenshots:
The screenshots will also be automatically generated when you stop the test, so there is no need to manually do it. Thanks to this Visual Website Optimizer’s new feature, now you never have to worry about remembering what were you exactly testing last month!
We hope this new feature helps you in archiving your A/B and multivariate tests. We have been thinking of extending this feature by including tiny thumbnails within VWO reports and also in future these screenshots will be used while generating a PDF report! Your feedback and feature suggestions on screenshot feature will be great.
We are proud to announce the integration of Visual Website Optimizer (VWO) with Unbounce, which is a tool for creating landing pages. If you are a VWO user and want to create a new landing page, you now know the tool we’d recommend.
The integration of the two tools is dead-simple and Unbounce has put together a post announcing why they recommend Visual Website Optimizer for multivariate testing.
Benefits of Integration
The main benefit, of course, is that using Unbounce you can effortlessly create new landing pages. So, if you want to start a new PPC campaign, for example, you won’t have to wait for your IT team to help you with that. Simply fire up Unbounce, create a new landing page, integrate it with Visual Website Optimizer and start optimizing it.
The main benefits of using Visual Website Optimizer with Unbounce is that you would be able to do the following for your landing pages created in Unbounce:
Read more about Visual Website Optimizer + Unbounce and watch an instructional video.
We just launched a cool new feature in Visual Website Optimizer: email notifications! VWO already has a feature which monitors your test in the background and disables loser variations or starts displaying winning variation to all site traffic automatically. This ensured that the test reduced risk of losing sales and conversions because losing variations get automatically disabled. Now, we have added email notifications to automatic monitoring so that any time a winner or loser variation is found, you get an email like the following:
UWe also did a quick update in conversion goals, so that you can now choose any goal as primary goal (which will be analyzed for crunching email notifications). So, now you can pick the conversion goal (note: you can add multiple conversion goals for each A/B test) you want to monitor, choose to disable losing variations for that goal and simply sit back and relax. You will get updates as soon as losing or winner variations are found!
It has been criticized, but it is always guaranteed to work. What is that we are talking about? Yes, you guessed it right: red buttons! No matter how many people consider such A/B tests as a trivial exercise, every now and then they have been demonstrated to increase conversions.
The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.
All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!
Here are different variations that were tested:
| With text buttons | With green buttons | With red buttons |
 |
 |
 |
The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)
As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.
The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.
In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.
Most A/B testing solutions, including Visual Website Optimizer, fall into a category known as client-side testing. What “client-side” means is that a variation of your test page is created (before displaying to the visitor) by doing manipulations on the browser using clever JavaScript. Your web server always sends the same default version of the page as it would normally do. Since the changes needed to display a variation happen on visitor browser (and not your server), this technology is called “client-side”.
There is another breed of testing frameworks known as “server-side” A/B testing. Using such frameworks, no modification happens on browser level. In fact, when the browser requests your page, a randomly picked version of your test page is sent from your server. While this means you need to involve your developers for implementing changes in your website code, it does give you unlimited flexibility of testing.
The server-side A/B testing framework for PHP that we officially support is (unsurprisingly) called phpA/B . It’s free to use (for personal websites). If you know PHP, see the documentation for this library. Just as VWO appeals to marketers because it lets them do A/B testing without making any code changes, we are sure phpA/B will appeal to developers who prefer to dive into code. The library provides integration with Google Analytics out-of-the-box. It also filters out all bots and shows them your control page so that it doesn’t affect your SEO efforts. If you are a Ruby on Rails developer, you may be interested in A/Bingo or Vanity.
Why are we supporting server-side testing frameworks? Aren’t they in competition with us? Not really! Our focus with Visual Website Optimizer is to make A/B and multivariate testing easier by taking technical hassles out of the picture. In fact, ideally, we want our users never to touch any code (be it HTML, JavaScript, CSS or PHP). So, VWO is suited for people who don’t want to rely on developers or IT team for doing even the simplest kind of testing (thereby lengthening the time involved in getting the test up and running).
On the other hand, server-side testing frameworks such as phpA/B are better suited to companies which want to tightly integrate A/B testing in their development process. This is in line with the upcoming “lean-startup” or “test-driven-development” scene.
Different companies, different needs, different A/B testing solutions! We hope you find the best fit for your needs.
A customer requested this feature, and we obliged! Visual Website Optimizer reports now come with a date picker to let a user see reports only for a specific date range (of course, in addition to the default aggregated report). See below how this date picker looks like:
Although test reports should always be analyzed in aggregate (to avoid introducing any statistical bias), this feature will come handy if you want to analyze following scenarios:
We are excited about this new feature, and it goes on to show that we are working towards making reports powerful and flexible to allow different kinds of analysis. Hope you too like the new feature!
Note: the date range doesn’t apply to clickmap and heatmap data. They are always shown in aggregate.
You Need A Budget (YNAB) is a user-friendly application for personal finance management. The company teaches people how to become awesome at managing their money by following a 4-Rule methodology. And they sell personal budgeting software that follows the methodology. The software is elegant, simple to use, and does just what you need it do – meaning you don’t bog down in unnecessary features that eventually make you want to quit (and pull your hair out).
Using Visual Website Optimizer, they wanted to increase the number of downloads for 7-day free trial of their personal budgeting application. For their first test, they chose to optimize product tour page. For software applications, the tour page is the one which takes up the job of convincing a casual visitor to try out the application. Motivated visitors will anyway download the application so it is the semi-interested visitors who need to be convinced. To optimize the tour page, YNAB chose to do a simple A/B split URL test with the conversion goal as downloading of the free trial.
They designed the variation with following objectives in mind:
As you will see below, the major change was including many more screenshots and hence not requiring visitor to click on different ‘categories’ to see what app can do. You see, semi-interested visitors are more likely to hit the back button than activity hunt for what your app can do. So, including all relevant functionality of the app in form of screenshots on a single page did the trick of convincing them to download the free trial. Here are the screenshots of variations:
Original (control)
Redesigned (variation) – 85% increase in downloads
The variation with more screenshots and lesser categories/links increased downloads by 85%. This result was statistically significant at 97% confidence, so YNAB has implemented it permanently for their tour page. In their own words, here is what they concluded from the test:
We simplified the page presentation and strengthened the call to action with a very convincing and authoritative quote.
Thankfully, like many other customers, they had great praise for Visual Website Optimizer:
Visual Website Optimizer was extremely valuable. We’ve been working with Google’s website optimizer for the past three years and we’re never going back. We were constantly having to jimmy-rig the script code, make sure it was not conflicting with Adwords code, or Google Analytics, etc. It was a big enough nightmare that we avoided testing as much as we could – something that cost us real money over that time obviously.
Are you wondering what is going to be the next page they are going to optimize? Homepage, of course! YNAB will optimize their complete funnel from homepage to product tour to payment. We wish YNAB best of luck for their future tests.
