and so should you...
Visual Website Optimizer API was privately announced to select users long time back but today we are happy to announce it to general public too. The API lets you pull complete test results into your internal dashboards. Results are returned for all test goals and include confidence levels, hits and conversions. amongst other metrics. The API also allows you to start and pause tests programmatically.
API interface is JSON and is currently available only by request. If you wish to use the API, email us at info@wingify.com and we will provide you an access token and documentation for using the API.
When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.
Background
Soocial is an online address book that helps you keep your phone, computer and online services contacts sane. The app syncs, merges and provides backups so that you are never stranded without the contact details you need. It works with over 500 phone models, webmail (like Gmail, Hotmail and Yahoo) and your Mac and Outlook. Soocial chose to use Visual Website Optimizer for their testing needs. Their first test was to increase the click-through rate on the homepage to the signup form.
What was tested
Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.
Results
As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.
Control: 14.5% conversion rate
Variation: 18.6% conversion rate
The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.
Why “It’s Free” Worked
When we asked Soocial why they thought the winning variation worked, this is what they had to say:
Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.
So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:
To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.
For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.
How valuable was Visual Website Optimizer for the A/B Test?
Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:
Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.
Key Takeaways
Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!
These days you have all sorts of trust badges available to put on your websites. Some are SSL badges, some are good business practices badges and some are even carbon neutral certification badges. But do these badges have any effect?
A Visual Website Optimizer user, Karsten Lund (a conversion specialist from Optimeria.com), recently carried out a six-month long A/B test to determine whether an E-mark badge (certifies ethical conduct of Danish businesses) impacts conversion rate. He wrote a detailed case study in Danish and we recommend you to read it here (if you understand the language). For those who prefer to read the case study in English, read this post instead.
What was tested?
Karsten tested the presence of E-mark (the eye-like icon in the image above) on an eCommerce store House of Kids. 50% visitors saw this badge and 50% didn’t. The test started producing great results right from the start but our team consulted Karsten to run the test longer so that he can be really sure about the significance of the results.
What was the result?
Yes, you read it right, a 32% increase in conversions with the E-mark badge. And Karsten ran the test long enough to make sure result was statistically significant.
Interesting nuggets
A lot of other users agree with him, though.
Grain of Salt
Even though this particular result has been fantastic, you shouldn’t adopt trust badges on your website without testing them first. Different types of websites/badges combinations may have different results. Elsewhere on Internet, some report an increase in sales, some report a decrease in sales. So, the best strategy is to test it first and then only implement it on your website.
We have released a new feature in Visual Website Optimizer which lets you host the JavaScript files on your own server. This was done because a lot of large corporations avoid including external JavaScript files from a third party. Even though our JavaScript is hosted on Amazon S3 and we have a fail-over strategy that guarantees zero downtime (even in case of our servers going down), some corporations feel much safer if they control the resources (JavaScript, CSS, etc.) used on their websites.
The latest feature allows you remove dependency on our server while the test is running. All the test settings, variation content and code needed to run the test will be stored on your server (as four separate JavaScript files). VWO server will only be used for logging hits and conversions (which is an asynchronous activity).
Hopefully, you like the freedom of hosting JavaScript files on your server! Let us know your comments and thoughts using the comment form below.
Visual Website Optimizer is finally out of beta! It has been five months since we first announced the beta, but with all new features and bug fixes etc. it feels like an incredibly short period.
We also announced the pricing plans. There has been a lot of research gone into it, did various iterations of it and now finally settled on the one you can see. Most of our beta users thought this pricing is fair, so we are happy with their reaction. You will notice that there are various paid plans suiting all kinds of businesses and agencies. Additionally, we are offering a 30 day unrestricted free trial for the tool.
Limited Time Launch Offer
We are offering tons (2x-5x) of additional visitors tested in all plans till 15th June 2010. That means if you buy a paid package now, you will get additional visitors every month in your account (even after the offer ends). We think it is great way to thank beta testers and other early users who contributed a lot in shaping the software.
Thank you all! The journey to rule A/B testing world has just started
There are tons of A/B testing articles on the Internet. In fact, Visual Website Optimizer blog is full of them. But let’s admit it, all those guides, tips, tricks ultimately start sounding similar to each other. I decided to write a post on tips that do not have a straightforward logic and are not popularly discussed on the Internet. In fact, people usually do the opposite of what I am going to discuss in this post.
I won’t paste links here but a quick search on Twitter will confirm that people love raving about the A/B test they are currently running. You should absolutely not do this. Apart from having no real benefit, you can actually skew your results big time. With an A/B test, you want to determine how well variations work on your usual website traffic. Normal site visitors and curious visitors looking for your new test are very different in their behavior – you don’t want to bias your results with non-conversions of all visitors whose motive is to actually see what is being A/B tested.
If you are expecting a spike in traffic, it is best to avoid testing because: a) majority of visitors in the traffic spike will not be like your typical website visitors so they are usually going to bias your results in a big way. Imagine for a moment that you are observing a 20% increase in conversions with variation A as compared to control. Now TechCrunch covers you and sends you bootloads of traffic. This traffic is known to be come-and-go type. So, your overall conversions will go down. Agreed that conversions will go down for all variations, however the difference between performance of variations will narrow down. If variation A was beating control by 20% earlier, after the traffic spike it will be down to 10% or so for no good reason. So, consider pausing your tests if you expect some major publicity.
A/B testing purists may disagree that you should set the test once and then let it complete, but I say you should definitely keep an eye while the test is running. Of course, the primary aim of the test is to ultimately increasing site conversion rate. But you don’t want to wait till the end and realize that you have lost many sales during the testing period. Consider disabling poorly performing variations and making variants of best performing ones in order to increase conversion rate even the test is running.
Those were 3 A/B testing tips. If you have any controversial A/B testing tip to share, do leave a comment here.
In the last two days, we have released a bunch of small, unrelated new features in Visual Website Optimizer. Here they are:
Hope you liked the new features! As always, if you have any feedback, leave a comment here or email us at info@wingify.com
We are starting a new series to profile testimonials and success stories of users who love Visual Website Optimizer. The idea of I Love VWO series came to us when one of our users, Melvin Ram, sent us a testimonial full of praises (without asking, how fantastic of him!). Following is what Melvin Ram from Volcanic Web Design Company sent us:
I love VWO because it’s frictionless. Within 1 hour of setting up an account, we had 2 tests running. Not planned, but running!
Now that we’re doing testing, we’re making smarter decisions because they are based on real data about the behavior of our prospects. We previously didn’t take the time to track this because we’re a relatively small company and with Google Website Optimizer, setup took way too much time. VWO has already prevented us from making a big mistake that we were planning on trying for a month. Instead, we were able to run a limited test and within a few days had enough data to see that our clever idea wasn’t such as a good idea after all.
Now there is no need for guessing or debating internally. We just run a test and see what really happens. We’re fully committed to testing & conversion optimization and VWO is our tool of choice because they are the only affordable solution that we know we’ll actually use all the time because it’s so damn easy.
Even though we’re a relatively small company, our ambitions are bold. With VWO, we know we’re outsmarting our competition instead of outspending them. That is why I love VWO!
If you too have a story or testimonial to share, please email it to love@wingify.com. We will definitely get it featured here.
Most of the times you create and run A/B tests on all your website visitors. But there may be cases when you would like to test your ideas only for a particular segment for visitors. For example, you may want to test if a stripped down layout of website works better for returning visitors. Or you may want to test if a Twitter-specific banner gets more clicks from visitors arriving from Twitter.
New feature: Segmentation
We are proud to announce that we have launched Segmentation feature in Visual Website Optimizer. Using this feature you can choose to run the test only for the visitors who qualify your segmentation conditions. Have a look at the following screenshot:
You can choose and combine any of the following conditions available for segmentation:
Through the seven conditions above, we have covered most of the segmentation scenarios. Is there any other condition that you think could be useful for segmentation?
Behavioral Targeting and Personalization Possibilities
This is the most interesting part of segmentation. Thanks to flexibility of running multiple tests on the same page, you can personalize landing pages according to visitor segment. For example, if you have a generic landing page for all your marketing campaigns, using segmentation you can show campaign specific headlines and images to different types of visitors. In other words, thanks to segmentation feature, you can target your visitors with segment specific landing page and test it against control if it really drives up conversions. (You can also do neat stuff like inserting search keywords into variation or personalizing page according to city or country. See Advanced code mode for details).
We are *VERY* excited about segmentation feature as it allows power users to run targeted tests and personalize experience for visitors using a super-easy WYSIWYG editor. If you have any ideas or suggestions, please leave a comment below.
Do you like the segmentation feature? Would you actually see a use-case for it on your site?
Subscribe blog via RSS
