Visual Website Optimizer

Sometimes it takes a radical redesign to get big results. ClickLab is a Brazilian agency specialized in Conversion Rate Optimization and they used Visual Website Optimizer to test a radical redesign of the primary page on WriteWork.com, a popular essay website for students.

They chose to focus on the page which receives most entrance traffic. The traffic is nearly exclusively organic and the page has been struggling with high bounce rates for a while. The objective was to increase engagement and get more people further down the funnel.

The Arsenal

After crunching user surveys, getting feedback from tools such as UserTesting.com, Fivesecondtest.com, Gazehawk.com and ConceptFeedback.com a radical redesign was developed and recently tested using Visual Website Optimizer.

Targeting A/B test to a specific visitor segment

One challenge was how to go about testing the new design. Pretty much everything was different, from header to footer. This means a user might land on the homepage, which has one look, then click on to the redesigned page, which has a different look.

Luckily this is easily solved in Visual Website Optimizer. Since the primary objective was to increase engagement on the landing page, the following segmentation trick was used:

By restricting the test to only the visitors landing on the page, engagement could be measured without having to worry about the design not being consistent throughout the site.

Design Variations

The original page didn’t communicate what the benefits of the service are. Following is how original design looked like:

After reviewing over 1,000 survey responses it became clear what the real benefits were and then these benefits were communicated much more directly in the variation. Secondly, social proof and various credibility indicators were introduced. Here is how variation looked like:

A/B test Results

The result of the test was that engagement more than doubled. One important thing to note is that engagement in Visual Website Optimizer (VWO) only measures clicks and not submissions of forms, for example using a search form (for this reason searches were measured separately).

The great thing about VWO is that you can measure multiple goals. This meant it’s possible to see exactly where the engagement was happening – the biggest increase was indeed clicks on the CTA. The four top buttons accounted for less 1% in the engagement increase.

The secondary objective was to get people to click on the primary CTA and here results increased by 144%.

This meant that more than twice as many people continued on to the payment page. More payment page views doesn’t automatically more payment, but a follow up test showed that payments went up by over 50%.

Actually, although the results were fantastic, they weren’t that surprised. After going over the surveys from users it became so clear that the original really did a terrible job at selling the service.

Lessons from the A/B test

What allowed them these fantastic results was that they started with the user in mind; doing everything they could to understand him/her. Understand their wants, their worries and their needs. They also looked very closely at the language used on the website. For example, WriteWork has always used the expression “overcome writer’s block”, but no users used these words. Instead users wanted to hear “get started” and “get inspiration”. So now they use that language on the website.

Role of Visual Website Optimizer

Jens Schriver from ClickLab gave Visual Website Optimizer a nice, short testimonial:

It was a breeze to setup this A/B test and segment it. We’ve used Google Website Optimizer many times in the past, but – if we can avoid it – we are not going back

Hope you liked this case study. Our online case study library has many more A/B split testing case studies. And if you’d like to do a similar A/B test on your website, we have a free 30 day (no obligation) trial of Visual Website Optimizer.

10 Comments »

I recently got interviewed on Unbounce blog as a conversion hero. In the interview, I shared few rules-of-thumb related to A/B and multivariate testing which you may find helpful. I developed these heuristics while observing and advising hundreds of tests created by Visual Website Optimizer users. So, in this post I will paraphrase and expand on some of the things I shared in the interview.

A/B or Multivariate, which test methodology to choose?

Three main criteria will help you in choosing the right methodology between A/B testing or Multivariate testing:

• Traffic on test page: MVT requires lots of traffic to get any significant results
• Design resources available: MVT requires less number of design resources
• Objectives of the test: MVT is used for optimizing existing design and A/B test is used for optimizing conversions by testing a completely new design

Quoting from the interview, here is an elaboration on these three factors:

The eligibility criteria for each method is traffic of course. You should not attempt to do MVT if you don’t have enough traffic on the site. But assuming traffic isn’t a constraint, MVT works best when you are hyper-optimizing. That is, when your aim is to squeeze the last drop of conversion rate juice from your existing design. On the other hand, A/B testing should be used if you want to test completely different designs and ideas. Ideally, an organization should do lots of MVT tests followed by a few large A/B tests.

MVT typically requires less design resources as compared to large scale A/B test changes. Moreover, as I said, if the objective is to optimizing existing design MVT (or single element change) is way to go. But if you want to do radical changes on the page (say layout change, theme change, etc.) you will go with A/B testing.

Best methodology to start with?

Undoubtedly, if you are just getting started with testing and conversion rate optimization, you should go ahead with a simple A/B test. Multivariate testing is a complex methodology and it is easy to draw erroneous conclusions. From the interview:

For the starters, I always recommend to start with small-step changes in order to truly appreciate the value of testing. Ideally, they should pick a sweet spot on their page (ideal candidates: call-to-action, headline and image) and optimize that by a simple A/B test. Only once they get the hang of the whole process, they should attempt MVT or large-scale A/B test.

What to test and what not to?

Of course, what you test on a page depends on the specific site and objectives of the test. But if you are looking for some rules-of-thumb on what are the most common elements on a page that can be tested, here are they:

• The King: Call-to-Action (your main button)
• The Queen: Headline
• Others: text copy, images, number of form fields, number of steps in funnel, required vs. optional steps, number of elements on page, amount of text on page, layout (left vs. right kind of tests)

As far as what not to test is concerned, it is best to avoid testing:

• Pricing: very risky, and potentially illegal. You shouldn’t offer the exact same service/product at different price-points.
• Trivial elements on site: every element being tested on a page should have a hypothesis on why you are including in the test. For example, without a specific reason, you shouldn’t add page elements (say a footer or header) in the test and expect conversion rate to improve magically! You need to be convinced that a particular site element has high chances of impacting conversion rate.

What kind of surprises can you expect while doing A/B testing?

Technically no winning variation in a test should be seen as a surprise because there had to be a specific hypothesis on why you included it in the test. Nevertheless sometimes one finds that the test results are contrary to what was expected. That is, a variation won hands-down when one such expected it to lose significantly (or vice versa). Here are a few (real-world) examples of such surprises:

A recent test was very surprising – in this test it was found out that removing a secure icon from the page actually increased conversions by 400%. Another surprising result was that by simply adding a human photo on a homepage, conversion rate can be potentially doubled.

One of the test results on our homepage goes really against the standard advice of having a ‘Signup’ button prominently features on homepage. We found that a ‘Signup’ button actually decreased eventual sign-up’s and ‘Watch a short video’ worked much better because after watching the video, visitors were sure of what they are signing up for. (We had a ‘Signup’ button on the video page, by the way).

I hope you liked the interview snippets. If you want to read the full interview, head over to Conversion Heroes Part 3: Split Testing – An Interview with Paras Chopra.

1 Comment »

A multivariate or A/B test doesn’t end when you stop it. It is important to retain the test if you decide to revisit it later. A good archive of past tests is a powerful information repository; an organization should know what it had previously tested and what the results were. The knowledge derived from previous tests can be of great use in deciding what to test in future. For any A/B or multivariate test, you should aim to record the following before completely stopping it:

• Original hypothesis of the test
• Test Results
• Discussion about test results
• Screenshots of variations that were tested

You will thank yourself for recording these four pieces of information. Nothing can be more frustrating than trying to figure out what exactly did you test a month ago! Fortunately, with Visual Website Optimizer, this process is mostly automatic. It always retains the test results and the only manual part, of course, is that you need to write down your thoughts in the Test Notes section.

New Feature: automatically generate screenshots of test variations

Earlier you had to depend on preview feature to see what did you test in previous tests. The preview feature will only work perfectly (and show original variations) if your test page didn’t change. But that is seldom true! (In fact, after a successful test, you will change your test page).

Now, the new feature allows you to automatically generate full-page screenshots of variations and then those screenshots are stored on our servers so that you can revisit what you were testing any time in future. In the preview screen, here is how you take the screenshots:

The screenshots will also be automatically generated when you stop the test, so there is no need to manually do it. Thanks to this Visual Website Optimizer‘s new feature, now you never have to worry about remembering what were you exactly testing last month!

We hope this new feature helps you in archiving your A/B and multivariate tests. We have been thinking of extending this feature by including tiny thumbnails within VWO reports and also in future these screenshots will be used while generating a PDF report! Your feedback and feature suggestions on screenshot feature will be great.

No Comments »