and so should you...
It is a myth that A/B testing can only be used for optimization and you need to fully establish a process optimizing it. In other words, many people think that A/B testing is best meant for big guys who already have an established conversion funnel, and now want to squeeze an extra 0.01% out of it. As a natural consequence to this logic, it may appear that A/B testing cannot be used at initial stages of a project (or startup) because there is nothing to optimize yet.
Wrong. There are actually two distinct uses of A/B testing: a) optimization; b) validation.
How to use A/B testing for Optimization?
As mentioned in the opening paragraph, this is the activity with which A/B testing is usually associated. Typical scenario: you are running an advertising campaign where you collect leads. You get 10 leads for every 100 visits and you realize at this conversion rate (10%), this activity isn’t profitable. So, you set out to optimize (or increase) the conversion rate. You test headline, product image, form length and a bazzilion different page elements. In the end, you manage to increase the conversion rate to 15%. Well done!
As you can see, with this A/B test, the objective was clear: optimize conversion rate for the activity one is currently doing. There are several Visual Website Optimizer case studies where the tool was used for optimization. Examples include increasing sales by 20% by testing page design and increasing sales by 6.3% by testing Buy Now button.
How to use A/B testing for Validation?
Unlike above where we answer what is the best way to do an activity, in this case A/B testing is used to answer which activity to do in the first place. Validation means coming up with different possibilities and testing which one works best. Typical scenario: you decide to publish a new whitepaper for gathering leads. However, you are unsure which whitepaper will be most interesting to your customers. To answer that question, you make 5 different landing pages for each whitepaper and setup an A/B test. After the test is over, you check which whitepaper garnered most interest (via pre-release signups or some other conversion metric). Notice that you aren’t optimizing the lead collection process here, rather you are testing which whitepaper will potentially collect most leads.
To give you some examples, there are several Visual Website Optimizer case studies where validation was carried out. Here is how a startup validated its positioning and here is how the appropriate position for a promotional message was tested.
Key Lesson
A/B testing isn’t just for big corporations who want to optimize the last drop out of their conversions. In fact, A/B testing is more useful at answering business questions. Want to know which product to launch next? Which feature to develop? Which new market to enter? A/B test it today!
What is your opinion about this? Did you ever use A/B test for validation, instead of optimization?
We are in the process of finalizing pricing for my startup Visual Website Optimizer, which is an A/B and Multivariate testing tool. As you can imagine, fixing price is one of the toughest decisions that a startup has to (inevitably) take. Once fixed, it could be extremely difficult to change it without annoying a lot of customers. We want to be extra sure that we don’t end up under- or over-pricing Visual Website Optimizer. So, how do we decide what to charge?
Asking beta users can be one of the strategies and we actually used that for VWO. However, it turned out to be not the best of our ideas because users actually correlate price with quality. Further, if a product is innovative (like VWO is), users aren’t able to rationally determine its price as they would do in established markets (read this research paper [PDF]). Hence, asking users what a new product should cost yields half-baked information as they have little or no reference points to determine the ideal price.
Another way to determine price is, of course, to look at what competitors charge. In the world of A/B testing, there is a lot of variation. While Google’s basic testing tool is free, Omniture and Webtrends solutions easily run into thousands of dollars per month. To be sure, VWO is not competing head-to-head with a free tool – it is a sure shot way to death for any startup. We are actually competing with pricey tools, at least in terms of functionality. In fact, our users’ feedback confirms our hypothesis that VWO is much better than any other A/B testing tool in the world. Does that mean, like enterprise tools, we should also charge thousands of dollars per month?
So, here we are: still undecided about the pricing. We don’t want to charge too high, neither we want to charge too less. To get better insights into pricing, I decided to research if there had been a successful startup in a complementary industry which was in a similar position. I was lucky enough to find Clicktale, a company which provides heatmaps and usability testing tools. Thanks to Archive.org, a great way to trace their progress over years is to look at their website and pricing evolution over time. (Note: I don’t have any insider information about their company nor I am affiliated to Clicktale in any manner. All material that I have used for research is publically available on the Internet).
Clicktale started in May, 2006 as a closed alpha. Within a month, they quickly transitioned to beta and did not exit from beta until May, 2007 (a full year in beta). However, they still remained invite only till July, 2007. Within this beta period they started taking feedback from their users on what they can pay for the tool. Read the thread titled ‘Pricing for Clicktale’ on their forum (dated Feb 14, 2007). Every reply on this thread is a must read, however the poll results summarize their users’ feedback on pricing:
All startups in beta take note from the poll above – chances are that more than half your beta users won’t like to pay for your product. In our survey too, many of VWO users either did not want to pay or wanted to pay a very small amount $9 or $15 per month.
In the forum thread above, two contrasting opinions are interesting:
One user thought that higher pricing is justified as the tool is a direct value addition to business. He also rightly pointed out that the target market is online business owners, who are already conditioned to pay for services. To quote him from the thread:
So $99 is really low priced. If someone cannot take the $99 or even the $49 ClickTale product, and use the information to at least create a positive ROI every month, they probably do not need the product in the first place.
Then there is a user who thought $49 was too pricey for him and $19 was too limited. He compared the offering with Google Analytics, which is free. To quote him from the thread:
As to Adam’s comments [the user who argued $99/month was fine] about pricing, I think he and I are not operating in the same world.. I use Analytics for free, and am paying $19/month for crazyegg. I don’t use enterprise-price-level solutions and I think probably a lot of your customers don’t either.. If you are going to be in the enterprise world, just hire a ton of people for sales & support and quadruple your prices..
This contrast is super-interesting to us because, like them, we are in exact same dilemma – which market to cater? Do we cater to small/medium and enterprise sized businesses who can pay for the service? Or do we cater to long tail of freelancers or tiny businesses who want the service for free or at an extremely low cost? As a startup, the decision of identifying target market pretty much determines our survival. A no brainer: price it too high and we don’t get enough customers. Price it too low and we don’t get enough revenue to cover costs.
Clicktale finally launched to public in August, 2007. Their pricing at the launch time is interesting:
The pricing is same as they had proposed in the forum, however interestingly they increased the page recordings per month (the limiting factor for different accounts) by 5x. In the forum post, they proposed 80 recordings/month for free account, while on launch they increased it to 400/month. This tells they listened to feedback from their users carefully and incorporated it into their final pricing. Take note of that, startups.
In December 2007 (about 1.5 years after they launched alpha), they secured venture capital and the pricing of their product remained the same for next 7 months. In August 2008, they bifurcated pricing plans into personal and business editions. In personal plans, they offered a super-cheap plan of $9/month (10 times cheaper than the existing $99 plan):
And for business edition, they introduced a super-expensive plan of $999/month (10 times costlier than existing $99 plan):
Unfortunately, Archive.org does not have any entry of their website after August 2008. I searched on Google to get a hint of what happened to their pricing after they launched personal and business plans. The latest change in pricing that I was able to track down is in Feb 2009. A blog post reviewing them reproduced their pricing chart:
You can see they dropped bifurcation of plans and decreased the number of choices significantly. They also dropped cheaper plans ($9, $49) and some expensive ones too ($490, $990). They have exact same pricing even today (as of April 2010).
The reason I invested time doing all this research is to help startups such as mine take an informed decision on how to price their products. It is not very obvious that a successful company such as Clicktale, within 2 years, transforms the highest priced plan at the time of launch into their lowest priced plan today. They tried all sorts of pricing plans and apparently found that businesses which use their service value it enough to pay at least $99/month.
As a startup about to get out of beta, what I read from this is that if you provide a product that your users value, don’t fall into trap of selling it at rock-bottom prices (in hopes of compensating it from large volume of users). Another lesson that I take from this evolution is that it is OK to change pricing if your current plans don’t turn out to be ideal.
If you have any feedback, comments or thoughts, please share them with me. I hope you enjoyed my investigative journalism 
Answer: none. Using Visual Website Optimizer on my blog, I tested the layout of sidebar to see if it has any effect on bounce rate. Particularly, I tested whether the sidebar in right (default on many blogs) works better than the sidebar on left. The answer for my case turned out to be that it doesn’t really matter. (Though it may differ for you – you should A/B test it, it is really quite simple). Here are the versions I tested:
Control – Sidebar on the right (click to expand)
Variation – Sidebar on the left (click to expand)
Both versions had surprisingly similar engagement rate of around 22%. Engagement rate is inverse of bounce rate, so in a way this means , irrespective of sidebar positioning, bounce rate of my blog remains 78% (quite high, by the way). If you want to see actual results, here they are (click to expand):
Since the test involved changing the layout of blog across all posts, if you are curious that this would have involved digging into Wordpress or PHP code – the answer is no. In fact, the took just 5 minutes to implement. Using the advanced code mode of VWO, all I did was to define the Left Sidebar variation with this CSS: #sidebar { float: left; }. Yes, just that! VWO took care of the rest.
WordPress and Drupal are the most popular CMSes on Earth as they power a lot of business and personal websites. These CMSes have been game changing because they enable anyone can setup and manage a website easily without ever touching HTML or messing up with technical issues. However, if you need to do A/B or multivariate tests on such CMSes, then it is not exactly a piece of cake. All the technical issues (inserting code, tagging page, etc.) associated with setting up a test completely defeat the spirit of CMS. Separation of content from code is the first place why one uses a CMS thus running an A/B test on it shouldn’t require any messing up with code.
Because we realize the pains associated with creating A/B tests on CMSes, we are proud to announce two new Visual Website Optimizer plugins for WordPress and Drupal. With these plugins, you do not need to touch even a single line of code on your website to create and run A/B or multivariate tests. The real power comes from the fact that if you have these plugins enabled you can create unlimited number of tests from VWO without any sort of page tagging. Simply activate the plugin and you are all set for increasing your sales and conversions.
WordPress Plugin Instructions
This plugin is made by Andy Bailey of CommentLuv. Thanks for your effort Andy!
Drupal Plugin Instructions
This plugin is made by Will Ronco of Awesome Software. Thanks Will!
We will be adding support for more CMSes soon. If you have a CMS and you want to experience the smoothest and easiest A/B testing on it, let us know. We will make a plugin for your CMS too.
I am extremely happy to compile a list of new features in VWO:
Hope you like all the new features.
With Visual Website Optimizer, we like to keep things extremely simple. However, don’t mistake simplicity of VWO for less number of features. It is the job of a good UI designer to present a tool in layered fashion: core essential features being obvious to most users, while letting advanced users use plethora of other features by uncovering options as and when needed. That is what we have done with the latest new feature in VWO. It is a feature a minority of advanced users will ever need, but we still added it to VWO for completeness sake.
Here is the new feature: now you can define wildcards in split URLs too. Allow us to explain. Suppose you want to split test your entire website (and not just a single page). Maybe you are testing different themes or maybe you are testing different sidebar location or testing different placement of ads. While with CSS testing this is anyway possible without the hassles of creating separate versions of your complete website, but let’s assume you have created two or more versions of your complete website.
Now the challenge with other (free or paid) testing tools (and until now with VWO) is that you can’t do split traffic on the whole website! You need to specify a single page, and the tool redirects the visitor to one of the variation URLs and determines which one works best. However, such a strategy won’t work for site-wide split test because a visitor can land on any site page, and not just the specified page.
So, what we have done is to allow using wildcards in test page URL and variation URL. Check out the screenshot below:
Do you see the wildcard (*) here? Using it in your test URL will redirect the visitor to any page of your website (example http://www.wingify.com/contact-us.php) to corresponding page on variation website (example http://www.wingify.com/b/contact-us.php).
What’s more exciting is that you can even use multiple wildcards. If you are running an exotic test, then perhaps your control will be http://*.example.com/* and your variation will be http://www.example.com/b/*/*. Corresponding * in variation URLs are replaced by what matches in the control.
Hope you like the new feature. Please let us know your feedback, ideas, comments or suggestions though the comment box below or email us at info@wingfiy.com
We get bored (and frustrated) if we think it has been long since we last stretched the limits of A/B testing. So, in order to kill boredom and fell happy, we added a new feature to Visual Website Optimizer. The new feature, called Browse Mode, allows you to browse your website inside the visual designer and design tests spanning multiple pages. It even allows you to interact with login or other forms, so that you can create tests for the locked in parts of your application.
Have a look at the following 2 minute screencast which demonstrates how to create an A/B test inside (Wordpress) application:
Other uses of browse mode:
Technical details on how we did it:
Handling login forms and sessions inside the designer actually proved to be little tricky because cookies were involved. Every time you login into an application, the server sends back a few text files (known as cookies) which your browser is required to send back to the server with each subsequent request. Browsers handle this seamlessly for you however since Visual Website Optimizer’s test designer acts as a proxy between your browser and your server, we cannot simply pass on that job to the browser. So, we ended up implementing the cookie storage functionality of browser inside the VWO itself. It was fun and quite satisfying when we finally made it work. Though there are some edge cases that we haven’t handled so in rare cases, the login functionality may not work for all websites and apps.
Feedback
Try out the new feature and let us know if you like it. We worked hard on this one!
In our mission to be the best A/B testing platform on earth, we have recently updated Visual Website Optimizer architecture. This blog post updates what has changed and why.
Prelude: phasing out of optional page tagging code
In addition to tag-less testing which required no code changes, VWO had optional code for people who liked to tag their pages with code (like other free testing tools). As we all know it, tagging is frustrating, cumbersome and error-prone. So, we have dropped it all together from VWO. No user used this feature and it only created confusion.
Meaner: multiple tests on a SINGLE page possible now
Until now you could run only one test per page. While this is the way it should actually be, some advanced users ran a site wide test (for example, on footer or sidebar) and then they wanted to test a specific page also. With old architecture, one could run only one test per page so VWO only ran the most recent test on that specific page. The old architecture also meant that if one of the site pages is a test goal, you couldn’t run a different test on that page. This was especially frustrating to users who ran multiple A/B tests across their funnels.
But ALL this has changed. You can now run multiple tests on a single page or run tests on a page which is a conversion goal for a different test. A note of warning: we actually discourage multiple tests on a single page because it jeopardizes statistical significance of those tests. It is best to avoid such a condition.
Faster: asynchronous content loading
Now onwards, the content of test variations will be loaded asynchronously without stalling browser. This means while VWO is fetching variation content, browser can keep loading your rest of the page. This feature will make the tests noticeably faster, especially if you had lots of content in your variations.
Leaner: 10% smaller JavaScript code
We used Google’s excellent Closure Compiler to shorten the code by 10%. The current size of shortened, gzipped code stands at about 26.5 KB. Most of this code is JQuery, which VWO uses quite heavily (because it so awesome!).
Smaller JavaScript file means less data to download for your visitors and hence faster tests.
Overall Thoughts
We are committed to faster loading tests for your visitors. Every millisecond of delay frustrates visitors and we perfectly understand the importance of page load time for your business. Rest assured our engineers are fanatics about optimization (in all senses of the word). Do let us know your feedback on the updates and if you have any ideas/suggestions on making the test loading even faster.
RIPT Apparel is a Chicago-based online retailer of one-of-a-kind designer tees and wearable art. They sell one new design by a new artist every day starting at midnight CST for $10 and it is only available for a period of 24 hours. After that period, the design rests in peace forever in the T-shirt graveyard and a new one takes over its place. The sense of urgency created by limited availability of the design can be used effectively to drive sales and that is what RIPT Apparel did in their first A/B test.
They had never done A/B testing before but were intrigued by the concept and wanted to give it a shot. The tool they selected for this purpose was Visual Website Optimizer. They tested their current buy button against a new buy button to determine which would have the best conversion rate. The conversion rate here being sales of their T-shirts.
The following is their control version (or default). Observe that the button isn’t the first thing to catch your attention as it blends well into the overall color scheme.
Original Version
They saw an opportunity for improvement, so they replaced their existing button with an attention-grabbing button with a different color scheme (green). See version A of the button below:
Version A – Control
Much to their surprise, they immediately saw a rise in sales. Ideally, they should have A/B tested the new version against the old but this was their first time and as you will read further they did a great job overall. Encouraged by the results, they went ahead and created another variation of the button. This time they decided to emphasize on the tactic that they should have used all along: emphasizing “limited 24-hour availability”. It has been demonstrated again and again that a sense of urgency drives up the conversion rate and this time too it proved to work best. Following version (B) of the button increased their site sales by 6.3% (notice only the button was changed – no new offers, no new products, no new policies).
Version B – 6.3% increase in sales
Pause for a moment and think what really happened in this case study. Someone (RIPT Apparel) with no previous experience in the field decides to A/B test, spends time researching what could work better, designs button variations, uses Visual Website Optimizer and sees a 6.3% rise in sales. No gimmicks, no new offers, no change in company strategy. Simply testing of sensible better variations of buy now button.
That 6.3% rise in sales will be hopefully permanent, so this one time effort would pay back RIPT Apparel many times over for time to come. Here is what they had to say after the test ended:
“According to Visual Website Optimizer we found our conversion rate improved a whopping +6.3%! We were astonished by this result. We had no idea that changing one button could result in an increase in sales and have that much of an impact. We wish we found this tool much earlier. “
It is important not to forget the role of the tool here. Visual Website Optimizer has been designed with a sole focus on ease of use, while avoiding technical hassles as much as possible. When we asked about the role of the tool in their first A/B test, RIPT Apparel said:
“The sales we have seen since changing and experimenting with our buy button have proven Visual Website Optimizer invaluable. I had some issues setting it up due to our shopping cart system and the team at Visual Website Optimizer walked me through step by step in order to set it up correctly.
As a result of this test I was able to go back to my partners and show them the results, which they were very happy with. Version B. is now live on our site and sales continue to rise.”
A great thing about tasting success with A/B testing is that it pumps you up to do even more A/B testing. So, RIPT Apparel are all set for their next phase of testing:
“We learned to test and test often. We think now that this test is over, we are going to try and test a few different color options, maybe even try some wording choices for our buy button.”
RIPT Apparel is a great company that successfully proved that A/B testing is not something that only Fortune 500 companies can do. Even small-medium sized online business can optimize and increase their sales by not mistaking A/B testing as a hyped-up fad. A/B testing works and it is addictive; one simply needs to try it at least once.
By the way, you should check out RIPT Apparel’s T-shirts – they are really quite cool!
Subscribe blog via RSS
