The Next Generation of Visual Website Optimizer is launching in April 2014 See What's Coming

Signups increased by 60% after actually removing the signup form

Posted in A/B Split Testing, Case Studies on November 18th, 2010

It is interesting how counter-intuitive A/B testing results can get. One of the best examples of that counter-intuitiveness is a recent A/B split test. This test was done by a Visual Website Optimizer customer Vendio, an Alibaba.com company that specializes in providing free e-Commerce stores to its merchants. They have special landing pages for the free store signup and the objective of this test was to increase signup conversion rate on one of those landing pages.

A/B test details

Their original landing page (control) combined marketing content and registration fields in an attempt to reduce the number of clicks for a successful registration. This is how it looked:

Original Landing Page (with embedded signup form)

Note that it uses the so-called “best-practice” of embedding the signup form in the landing page itself. They had long been using the layout of the original page because as a best practice they presumed that reducing the number of clicks for a registration increased conversion rate. Although the page was performing relatively well, they wanted to make sure the included registration fields weren’t too aggressive or limiting in any manner.

So, the variation that they tested had somewhat similar imagery and content but the page didn’t include any registration fields and had slightly different styling. Clicking on ‘Signup Now’ button simply took the visitor to a page with signup form. In other words, Vendio added an extra step in their conversion funnel. Not a smart move, huh? This is how variation looked like:

Variation (no signup form) – 60% increase in signups!

A/B test results

Guess what? The page without the registration fields performed better – much better – to the tune of a 60% increase in conversions! Here is what Vendio had to say about the results:

Best practices are NOT always true! It’s still hard to believe, but the numbers don’t lie.

So, that’s the biggest lesson here and it is worth repeating: “Best practices are NOT always true”. If you do changes on your website or landing pages without A/B testing them, you are actually flying in the dark. Another lesson here is that it is it is worth testing radically different ideas – which, on the first glance, may appear not-so-smart (like removing signup form from the landing page!).

Testimonial about Visual Website Optimizer

After the successful test, Vendio gave us a great testimonial:

Visual Website Optimizer was extraordinarily valuable. It was far easier to use than other solutions (some of which we couldn’t even get to work properly after implementation). Without VWO we would still be missing out on all those registrations, and we are continuing to see great and other surprising results with VWO.

Paras Chopra

CEO and Founder of Wingify by the day, startups, marketing and analytics enthusiast by the afternoon, and a nihilist philosopher/writer by the evening!

The Complete A/B Testing Guide

Know all that you need to get started:

  • What is A/B Testing?
  • Is it compatible with SEO?
  • How to start your first A/B test?
Show Me The Guide


Tags

43 Comments
Nick
November 18, 2010

It’s not anti-intuitive for me at all. I much prefer the second version. The first one looks “spammy”, just like one of those “parked” domains websites, that you only need one glance to spot.

I don’t know why, though.

Gregory
November 18, 2010

The first version makes the signup form too prominent while the second version is showing what the products is before trying to make me sign up.

I’m pretty sure that you can keep the signup form on the same page with a solid design like the second version that would make the signup form less prominent on the page.

In one sentence: show me what you sell and let me decide what to do. That’s the best practice.

Joe
November 18, 2010

I think it’s pretty obvious, the 2nd version has a bigger title text – they see exactly what it is straight away. That title text contains the word ‘free’.

Janusz
November 18, 2010

Don’t you think that part of the reason is actually that on the second layout 100% Free is much more visible than on the first one? I would guess that the whether customer sees 100% free promotion makes much bigger impact on conversion then moving the form from one page to another.

Paras Chopra
November 18, 2010

@Janusz: yes, a big reason for why variation performed well is because it has large “Free E-commerce store” written in the headline. Moreover, in control page, the form competed for attention with product description. In variation, the description and the pitch gets full attention of the visitor.

Mark
November 18, 2010

@Paras: “Moreover, in control page, the form competed for attention with product description.”

..Actually, I would say the product description competed for attention with the Form. Since reading is done left to right and dark bold colors tend to attract focus, my guess is that users focused too much on the sign-up form and were less inclined to spend additional time reading the messaging content.

I wonder what the test results would have shown if the form was moved to the right or bottom of the page and different colors were used.

…maybe the best practice is sound but the execution was flawed….

Ian
November 18, 2010

This is not a proper A/B test. There are too many uncontrolled variables.

All that you can say is that the second design resulted in 60% more conversions.

This you can take as an indication that the second design is more successful, you can in no way whatsoever conclude that it is because of the sign-up form.

It may be because of the sign-up form, it may be because of the different layout, it may be because of the big “100% free”, it may be because the people involved just happened to not like blue.

The only way to come to a proper conclusion is to run an A/B test on two designs for which the ONLY difference is the feature you’re investigating.

Paras Chopra
November 18, 2010

@Mark: There are so many variables affecting conversions that it is hard to tell (without testing) what could have worked better. Yes, probably shifting signup form to the left and toning down the color would have worked. But who knows unless we do an actual A/B test?

Ian
November 18, 2010

(Just to be clear.. I do actually think that it is because of the signup form, they’re ensuring that the first impression of the page is always the sales pitch rather than an intimidating form, they don’t see the form until they are already bought into the idea. However it is quite wrong to claim that this A/B test provides evidence of that)

Paras Chopra
November 18, 2010

@Ian: yes, it is hard to say the *exact* factor which resulted in better conversions. In an ideal world, you would vary one factor at a time. But most organizations don’t have luxury of time and traffic. So they have to take this approach. But you are right we can only guess that removal of signup form is the cause but cannot be certain because the test provides no evidence for it.

Ian
November 18, 2010

Yep – for your purposes it’s not too relevant, as you’ve demonstrated that the second design is the one to go for. Regardless of whether you have the resources available to investigate the exact reasons you have still arrived at a more successful product.

However it’s very relevant for the purposes of this article, which is obviously false and misleading both in title and content.

Ian
November 18, 2010

Ironically, it’s precisely because of the wrong conclusions being drawn and evidence falsely attributed that you end up with the kind of incorrect best practices that the article is referencing.

Ellery
November 18, 2010

“Spammy” is the first thing that came to mind when I saw the first version. I think people want to decide when they’ll like to see the registration form. Also, after clicking to link to sign up, the user has already made some sort of commitment and is more likely to follow through.

dotjinks
November 18, 2010

“Don’t give them what you want them to have, give them what they want and when they want it.” Well, what I really mean is that you at the very least should build your funnel to make them feel like it’s all their choice!

Huh? You mean the way Google has been doing this with their minimal sign up pages (for many years now) actually makes sense?

Troy
November 18, 2010

Great topic! Never knew.

I like the 2nd one, because it concentrates on telling me the value add, before wasting space on a sign up form, before I understand why I need it.

http://tech.rawsignal.com

Justin Germino
November 18, 2010

Honestly the 2nd page looks much more professional and I would be more inclined to click the sign up button based on the design changes.

Lol
November 18, 2010

worst a/b testing ever. way too many variables.

Mike
November 18, 2010

The page that had a higher sign-up rate seems to funnel the eye down more to the sign-up button. The first page would probably make most people click off before even reading anything because no one wants to sign-up for anything anymore unless they feel they have already be locked in somewhat. Great case-study though and teaches everyone to test everything.

miles
November 19, 2010

There is no such thing as “best practices”.

Anyone who uses this term is trying to convince you of something. And in my experience that person is usually ignoring the realities of the local landscape.

Always know your customers.

jcl
November 19, 2010

Form B does not perform better because it is less ‘pitchy’ -

The act of clicking on SignUp psychologically commits the user to fill out a form.

The original page had to call to action for the form . Plus the form was on the left, people read left to right and they need the pitch first before they sign up. The first page was just BAD…no wonder you saw such a large change.

jcl
November 19, 2010

*NO call to action

[...] –Signups increased by 60% — after removing the signup form! Yet another KSU (Keystrokes per Sign-up) [...]

Jordi
November 22, 2010

Interesting article,

That was one of the first experiments we created when we started to use A/B testings, and really surprising for us it showed interesting cultural differences across the markets where we operated.

As you have said, our conclusion was “never assume the best practices are true”. For all of you operating across different countries it reinforces the needing of a specific strategy of testing across countries.

Very good article, thanks for writting it.

jordi

micah solomon
November 25, 2010

To do a/b testing, you should change ONE variable. This abcdefghijk/lmnopqrs… testing.
the “no credit card required” language alone in the first layout looks creepy, just getting that out alone may well have made all the difference, but we’ll never know because of all the variables changed.

Brett
December 2, 2010

Funny…

If you look up their site and click on the Adwords ad, they are STILL using the optin form.

Maybe this article isn’t all it’s cracked up to be.

[...] Visual Website Optimizer A/B Test [...]

Marc
February 11, 2011

flawed spit test..

come on guys!

it’s not an apple to apple comparison.

test your new design with and without a sign up form below…

sign up will win.

really poor thought process on this one…

Sean
February 16, 2011

Agree with Marc.

It’s not an apple-to-apple comparison.

My opinion? The headline (100% FREE Ecommerce Website) is more prominent in the formless design.
The form is not a magic deterrent.

Design 1 “leads” with the form and not the value proposition. It focuses on “SIGN UP” … but for what?

In design 2, the value proposition is more prominent and the user feels confident and empowered (informed) — so they sign up.

Still interesting.

[...] englische Originalversion dieses Artikels ist hier zu finden. Die hier zu lesende, deutsche Version wurde mit Zustimmung des Autors und Gründers des [...]

Trevor Adams
May 11, 2011

I think “best practices” are constantly evolving and A/B testing is one of the most important methods to ensure that what you think is “best practice” is actually the case right now. I agree with others that the first does seem to “sales” but the second is more informative.

[...] It’s important to test because you can never be certain of your customers’ behavior. Vendio, an e-commerce business was able to improve signups by 60% in a very counter-intuitive way: by removing the sign-up form from their landing page! You can read about this test in detail. [...]

[...] reading this case study, I noticed one big difference between the two variations of this A/B test.  It was for Vendio, and [...]

[...] lot of that coincides with putting a sign up form on your landing page. Well there has also been a few studies that show removing the sign up form can actually benefit your users just as much as putting the [...]

Days Spoilers
September 20, 2011

I’m just learning about testing and landing pages and I’m a bit confused. How much of the site do I have to change? If I change a lot, how do I know which change actually made a difference?

[...] at the following screenshots from a landing-page case study by Vendio. Which had the higher conversion [...]

[...] at the following screenshots from a landing-page case study by Vendio. Which had the higher conversion [...]

Nicolai Christensen
December 21, 2011

Interesting test – though it is not the first time it is seen that asking for too much in the begining of a process leads to lower conversion..

[...] die ihr Wissen als unantastbar verkaufen, sind nur mit Vorsicht zu genießen.(via VWO) Über Jörg Dennis Krüger Jörg Dennis Krüger beschreibt sich selbst als „versessen auf [...]

[...] You might be surprised at how much time and work goes into making a web design simple.  Rather than looking for ways to fit more stuff in, simple websites strip away the excess to deliver direct, concise messages in a limited screen-space.   What you may not realize is that taking the “less is more” approach to web design has some added benefits.  Beyond delivering faster-loading, easier-to-maintain websites, simpler sites also have a dramatic effect on increasing conversion rates. [...]

[...] case study describes how removing a signup form (from landing page and adding it in later steps) increased conversions by 60%. Another example is of our own homepage: when we removed the form and converted it into a popup, [...]

[...] Signups increased by 60% after actually removing the signup form « I love split testing – Visual … [...]

[...] Signups Increased by 60% After Actually Removing the Signup Form [...]

Laura Hamilton
December 27, 2013

Great post. I am a huge fan of data-driven design.

I do think, however, that you changed multiple variables at once in this A/B test. So there is a little bit of confusion about what the effect of the signup-form removal was vs. the effect of the other changes.

Leave a comment
Required
Required - not published


1 + = four

Notify me of followup comments via e-mail.

RSS feed for comments on this post. TrackBack URL

I ♥ Split Testing Blog


Stay up to date

Recent Posts

Write for this blog!

We accept high quality articles related to A/B testing and online marketing. Send your proposals to contribute@wingify.com