and so should you...
Well, OK, the post is not about exactly the kind of hackers who do nasty stuff, breaking into computers and systems. This post is rather a compilation of best discussions on A/B testing at my favorite online hang out: Hacker News. That forum is a must visit if you can relate to startups, technology, science and culture, all at once. (For those already on Hacker News, I’m paraschopra there).
On Hacker News (HN), A/B testing has become a somewhat regular discussion topic and the practical advice that you can get there is simply unmatched. Unlike regular blog posts and articles full of opinions, on HN you actually get to listen from people who have been doing A/B testing since ages (patio11) or people who have tremendous background in A/B testing (btilly).
In this post I will list some of the most interesting HN links and discussions on A/B testing. Make sure you check out discussions on each one of them. HN discussions contain far more information and advice than the actual link itself. (Warning: following links and discussions will yield value only if you spend time digesting them – simply bookmarking this post and then doing nothing about it won’t help you become an A/B testing ninja)
General Discussions:
Discussions on Case Studies:
I will keep this list updated as more interesting discussions on A/B testing crop up on HN. The forum is not just about A/B testing, there is a lot you can learn and discuss on Hacker News. So my recommendation to you would to start hanging out there, participating on all the interesting discussions.
You know what is the biggest backend challenge doing A/B split tests on-the-fly? Visual Website Optimizer (proudly) claims that you only need to add a small JavaScript code snippet once and then create unlimited tests from the interface without touching your website ever again. Since the code is added site-wide and not just the pages where split tests are running, the biggest challenge is to detect from which page a request has come and then to decide which test should run on the URL (if at all any). And given the idiosyncrasies of HTTP world (as you will see), all this needs to be done in real-time with every request coming in (currently about 200,000/day) within a couple of milliseconds (so that the effect on page load is minimal).
It is definitely an interesting challenge to solve and we are continously tackling different parts of it. Today, we rolled out a number of updates; most of which are related to the backend and URL matching. Lest you miss our hard work, I thought of documenting them in this blog post. Following are some of the major updates:
Massive boost in response time
We upgraded our servers (doubled the capacity) and did some backend optimizations (added new database indexes, removed some query time sorts). Result: the response time for displaying a test on your website cut to half. Proof: here is one the users tweeting about the optimizations.
larsgundersen “http://visualwebsiteoptimizer.com just did some speed optimization & server upgrades! Now the response time rocks!”
With S3 hosting of JavaScript and now backend optimizations, the total response time has been brought under 350 ms. Aim is to reduce it even further, because I know that response time matters a lot as far as visitor satisfaction is concerned. However, even if we do massive speedups, the real culprit for load time always remains the latency.
Tests work with AdWords or other referral traffic out of the box
Earlier, tests defined for http://www.example.com/page.html did not work by default for traffic coming AdWords (or other campaigns) because Google sticks some tracking parameters to the URL (making the resulting URL as http://www.example.com/page.html?utm_campaign=XXX). Visual Website Optimizer treated these two URLs as separate and hence the test defined for former URL didn’t work if the visitor arrived on the latter URL. A hack to make it work for both the URLs was to use the wildcard * (that matches with any character) so that http://www.example.com/page.html* matched with both the URLs. But, it was a messy way to do something which should work by default.
So, the good news is that we modified our URL matching algorithm to make it work out-of-the-box. Now irrespective of presence of tracking (or query) parameters in the URL, test would work for all visitors. (This wasn’t as straightforward as it seems because the wildcards still work and you can still define tests containing query parameters that would only work if there are query parameters in the URL).
To “www” or not to “www”, that is the question
A common usability issue earlier was that people defined tests for http://www.example.com/ but their website actually was http://example.com/ – this confused our poor, little URL matching engine (not because we cannot treat both URLs as one, but we don’t want to; reason being technically both URLs are different and even search engines treat them as different URLs). So, what we do now is smart: when you create a test we access the URL entered, follow all redirects (as a browser would) and then see what is resulting URL. In lay man terms, Visual Website Optimizer will define the test on URL that actually exists on your website, not what you have entered in the wizard. In super lay man terms, you don’t have to worry a lot about how exactly you are entering the URL. (However, you still need to be careful about how you enter URLs for conversion goals – those are case sensitive and no URL magic happens for it)
Oh gawd, the curse of “index.php”
If a martian arrived on Earth and you asked him whether these strings are same: http://www.example.com/ and http://www.example.com/index.php – what will be his answer? Now, if the super-smart martian can get this answer wrong, how can our poor little URL matching engine can get it right?
The world of HTTP is complex and confusing – replete with numerous special cases like the above two URLs resolving to the same location. Earlier our URL matching engine thought those were two different locations but now they are treated as one – so your tests will work on both the URLs. (Other special cases include common homepage file names index.htm, index.html, home.html, default.aspx, etc.). Essentially, you don’t have to do any hacks or changes to your website, “index.php” or no “index.php” test would work for both the cases.
Detection of overlapping tests
As a policy, Visual Website Optimizer doesn’t allow overlapping tests – that is, while you can run multiple tests on a website at once, you cannot run multiple tests on the same webpage. But this wasn’t very obvious so users usually end up creating multiple tests on their homepage and then complaining that they only see one of them running. With this update, while creating a test, you will get an alert if you are trying to create a test on a page where an existing test is already running.
Remember me?
A much needed, remember me functionality was added so that you don’t have to login again and again for check test results. Session timeouts and the need for logging in again was a common hassle, now it isn’t 
I will be happy to discuss if you have any specific questions on the URL matching engine, backend optimizations or other updates. Simply leave a comment here or email me at paras@wingify.com
AquaSoft was founded in 1999 and is a pioneer in photo presentation software. Their products cover a wide range from slide show creation, desktop publishing, photo books, etc. They recently candidly shared details from their latest A/B testing project, where they used Visual Website Optimizer for increasing sales from their eCommerce site by 20%. Here are the details on the test as told by AquaSoft:
Question: What was the conversion goal of the test?
Answer: The goal was to increase the sales on our product overview page. This page already had a higher conversion rate than the rest of our pages because people who visit this page already are interested in buying.
Question: On which page did you run the test?
Answer: Shop Overview (listing of all articles).
Question: Which part of page did you select for the test and what variations did you test?
Answer:
Original version (click to expand)
Redesigned version (click to expand)
Question: Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Answer:
Note that we did not add any new feature nor the guarantee was new. It was just not described on the old page.
Question: What results did you get? Were you surprised by the results?
Answer: Improvements started with about 50% but settled on about 35-40% after some time. I was very happy with the result but didn’t trust it. What was measured by VWO were the clicks into the shopping cart, not the actual sales.
So I measured the sales with the funnel visualization of Google Analytics which showed an increase of 17.7% of actual sales. It also proved that the number of clicks into the shopping cart were about the same what VWO had measured. It shows that the bounce rate within the shopping cart was somewhat higher in the winning combination but the overall increased click rate outtakes the bounce rate.
In the second phase, we removed some whitespace and moved the product name above the box shots and gained another 10% increase in clicks and the final increase in sales settled at about 20%.
Question: Any lessons which can be derived from your test?
Answer: Don’t think your website is good as it is. Always test, always improve. It’s a slow process but worth doing.
Our concrete test shows that a clear modern design improve sales. Especially in the sales process trust building is very important.
Question: How valuable was Visual Website Optimizer for this test?
Answer: Is was so incredible easy to create the test (A/B split test in this case) that it takes nearly no time. Because we use an external shopping cart we can not measure in the cart itself so the feature of VWO tracking a click on a link without loading the page made it possible to run the test at all.
A picture is worth a thousand words. And a video is worth laboring through hundreds of documentation pages and forum threads. The following video shows how to create a multivariate test using Visual Website Optimizer in well under 4 minutes. The short tutorial is a must watch if you have heard about multivariate testing in articles/blogs; and always wondered what exactly is it. Even if you haven’t heard about multivariate testing, don’t worry – this tutorial with demystify that. Basically, multivariate testing is a technique where you vary several elements on your page to see which version results in maximum conversions (that is, form submissions in this case). A successful multivariate test can often result in 50-100% increase in conversions (case study to come out soon).
Without any further blabbering, here is your video tutorial on multivariate testing (click Play to start the video):
Update: for faster response time and better reliability, we have moved to Amazon CloudFront. This post is outdated, you should read this post instead.
During the last server outage, some of the websites where Visual Website Optimizer was running slowed down because our servers went down for a while. Even though we have upgraded the servers and installed server monitoring, possibility of the worst scenario concerns me a lot. Having one’s JavaScript code included in hundreds and thousands of websites is a HUGE responsibility. Slowing down of our servers means slowing down of customers’ websites. I perfectly realize that responsibility – that is why we chose to use extremely reliable Amazon S3 for hosting the JavaScript files. What this means for you is simple – in rare case of Visual Website Optimizer servers being down, your website will continue to function normally without any downtime. Additionally, hosting JavaScript files on Amazon will speed up the loading time because bandwidth is not a constraint with S3.
The JavaScript code snippet has been updated in all accounts of Visual Website Optimizer. It is highly recommended that you update the code included in your website with the new code to take advantage of this new feature. The new code is available in the test reports, ‘Add code to your website’ section.
For techies here is how we are using Amazon S3. VWO code consists of two parts: a) static JavaScript file; b) dynamic JavaScript file to determines which test is to be run on the current page and what variation to display. While we can host static JS file on S3, dynamic JS file needs to fetched from our servers. So, in presence of a dynamic file how do we ensure zero downtime? Well, the servers where dynamic JS file is hosted are monitored every single minute from two different locations. If monitoring script determines that the main servers are down, it uses S3 API to update the static JS file. The updated JS files signals the JavaScript code snippet that dynamic JS file isn’t available hence shouldn’t be loaded. Effectively this ensures minimal downtime even when external dynamic JS is included. Now, the only case when users’ website can slow down is when Amazon S3 itself goes down (which is unlikely since they are bound by SLAs).
It is #1 feature requested and here you have it. This blog post introduces the new plugin for the Google Analytics. Using the plugin, you can easily benchmark multiple metrics across different variations. For example, you may find out that while your winning variation has reduced bounce rate significantly, it may actually be reducing average visitor time on site as well. Viewing results from multiple perspectives allows one to getter a better idea on the expected behavior of winning variation.
For integrating Google Analytics with Visual Website Optimizer, you will have to sandwich a snippet of code (in bold font below) in your existing Google Analytics tracking code (most likely present at the bottom of your website HTML):
<script type="text/javascript"> var pageTracker = _gat._getTracker("XX-XXXXXX-X"); pageTracker._initData(); if (typeof(_vis_opt_GA_track) == "function") { _vis_opt_GA_track(); } pageTracker._trackPageview(); </script>
The process for Asynchronous code snippet is similar:
_gaq.push(['_setAccount', 'XX-XXXXXX-X']); if(typeof(_vis_opt_GA_track) == "function") { _vis_opt_GA_track(); } _gaq.push(['_trackPageview']);
In other words, the code in bold (javascript function _vis_opt_GA_track()) needs to be added just before the Google Analytics code snippet _trackPageview
After you have added the code, the integration is done. No more settings, no more configuration. For all tests, the Google Analytics intergration will start automatically.
To see A/B test reports in Google Analytics, click on Visitors in the sidebar and then click on Custom Variables report. Then click on the test id (available from VWO reports page) corresponding to the test you want to see a report for. An example of the report name in GA would be “VWO-2” which means it is a report corresponding to test with id 2. In the report you can see goal values for different variations plus you can also see different metrics such as bounce rate, time on site, return visit, pages/visit, etc. corresponding to the variations. The reports look something like the following screenshot (click to expand):
As you can see, different variations have different bounce rate and time on site. If we had configured goals in GA for this site, we could have tracked performance of variations on multiple goals as well. We can also define an advanced segment for a variation to do any kind of complex analysis that GA allows us to do. How cool!
Before you get too excited, please note following important points regarding the plugin:
Technical note:VWO uses GA’s 4th custom variable slot by default. If you would like to change it (either because you already use if for a different purpose or you want to track multiple tests in GA), add the following code before any GA code:
<script type="text/javascript"> var _vis_opt_GA_slot = YOUR_CUSTOM_SLOT; // a number between 1-5 </script>
Leave a comment here if you cannot get the plugin to work for you or you have trouble understanding what code to add or where to add.
This post is a compilation of best A/B split testing case studies available on the Internet. In a way, this compilation proves that: a) A/B split testing really works; b) it doesn’t take much effort (hello Visual Website Optimizer); c) you should be doing it right now! I think case studies are an interesting way to learn about a new domain. So, consider this post as a starting point for the wonderful world of A/B split testing. Let’s start the list of case studies which you should definitely read today.
Writing Decisions: Headline tests on the Highrise signup page
37Signals did an A/B test on the headline of their pricing page. What they found was that “30 day Free Trial on All Accounts” had 30% more signups than the original “Start a Highrise Account”.:
Lesson: in a headline, the word “free” coupled with specific benefits (“30 day trial” v/s “start an account”) can do wonders.
Dustin Curtis’ “You should follow me on Twitter here”
The much hyped split testing case which involved testing of multiple different versions of call to text for twitter followers. Dustin found out that “You should follow me on Twitter here” worked 173% better than his control text “I’m on Twitter”. Though his results were fantastic, I recently wrote a post titled “You should NOT follow me on Twitter” arguing why you should be wary of applying this results without testing on your website.
Lesson: persuasive call to action “you should follow” proves to be much better than a passive call to action “i’m on twitter” (well, why would a visitor care).
A simpler variation which had less options for a visitor to choose from resulted in 20% increase in conversions. The winning version was also much easier on eyes (in terms of details and text) as compared to the control.
Lesson: too many options on a page usually overwhelms the visitor and s/he simply bounces off the page.
A/B Test Case Study: Single Page vs. Multi-Step Checkout
Reducing a two page checkout process to one page checkout increased conversions by 20%. Though overall increase in sales and orders didn’t increase much, checkout % did increase.
Lesson: lesser the steps you have in your funnel, higher will be your conversion rate. A lot of visitors drop off at each stage of conversion funnel so try to shorten it as much as possible (ideally to one page).
How To Increase Your Adsense Earnings 94% Overnight
Sensationalist headline but interesting results. hey tried different combinations – ads in left v/s right; ads top v/s bottom; test ads v/s image ads. Not sure if the results are statistically significant, but definitely a good case study.
Lesson: proves that A/B testing not just works for product/sales websites, but can work for advertisement supported pages too.
Marketing Experiments Response Capture case study – triple digit increase in conversions
They demonstrated a 258% in opt-in rate. Focus was to remove all distractions and require the visitor to only provide email address. For completing his/her complete profile, the landing page incited the visitors with an Amazon gift card (which was again split tested).
Lesson: require the visitor to input minimal amount of information on the landing page, you can always ask for more information later (it related is the concept of sunk cost).
How we increased the conversion rate of Voices.com by over 400%
In addition to A/B split testing, they used a lot of other tools/techniques – clickmaps/user feedback/usability testing. The final result was an increase in 400% increase in conversions.
Lesson: simplicity; social-proof; role of video in conversions; clarity in design
Skype Boosts Homepage Conversion Rate with Radical Simplicity
Skype tested their homepage for 3 different variations. Key difference between the three versions was the level of details presented on the page. Guess which variation won the test? The simplest variation of all (see image above) saw an increase of 5% in downloads. The figure may not look like much of an improvement but when you consider that Skype may be getting tens of thousands of downloads each day, this increase becomes quite large and totally worth-it spending thousands of dollars on doing the A/B test. (But, hey, you don’t have too – after all, Visual Website Optimizer is free to signup).
Lesson: simplicity, simplicity and some more simplicity
Do you know of any other interesting A/B split testing case studies which I should add in the post? Perhaps you may want to take help of A/B ideafox (a search engine for A/B split testing case studies).
You just spent $5,000 (or more) on a designer doing an overhaul of your website. And you also got a fancy enterprise web analytics package for more than $10,000 per month. Plus you have an army of web analysts in-house who churn out reports night and day to take your online business to the next level. But you know at the back of your mind that there is still something missing that numbers or expert opinion cannot capture.
What is that nagging thought? Yes, that’s right: it is the need for feedback from *real* people. “Usability testing”, as the industry likes to call it. It is a process wherein you observe people browsing your website and get feedback from them on how was their experience. Unlike numbers, real people have feelings. They get frustrated when they can’t find the Buy button. They get annoyed when the contact us link is hidden. They get confused when you present them with plethora of options. The real value of usability testing is unlocked when you start observing patterns in the responses. For example, if 90% of the users found it difficult to find out if your software is free to download then you should better fix that issue. Such insights are not rare when you do usability testing and are potentially worth thousands of dollars.
But usability testing is expensive. Proper usability testing has been limited to big companies who spend hundreds or thousands of dollars on it. And they do this activity because it is worth it. However for a small website owner, spending such a large amount of money isn’t possible.
Enter Feedback Army. Powered by Mechanical Turk, it lets you get feedback from 10 real people for just $10. What you simply do is ask people to visit your website and provide answer to questions related to their visit. I wanted to give it a shot for Visual Website Optimizer, and here is what I asked them to answer (based on their visit):
The responses were very interesting. For example, I always knew back of my mind the text-heaviness of the homepage but the answers from real users confirmed it. Take a look on what they had to say on the homepage:
It’s a bit confusing that in all areas the fonts are around the same size, not emphasizing on a certain more important zone. Simply put, you don’t know where to begin reading it.
When I read the text on the website, it was not really clear to me, still a little abstract. I would like to use more concrete terms. Less text on the page, there is a lot of information on it.
The technical terms are very confusing.
I would suggest reorganizing the information available on the landing page. With all that information at once it’s difficult to glean a simple understanding.
This was obviously a big insight. I now know the homepage needs a redesign to make it less confusing and less technical. (But designing homepage is always a challenge – however the issue here is to identify opportunities for improvement.)
On the flip side, what works on the homepage was also evident in the responses:
I watched the youtube video and that made it more clear.
The demo video in the features page is working well.
Not just only feedback, few responses actually gave direct suggestions for improvement. Take a look here:
I would like to use more concrete terms. Like one slogan: Alter your website and see the impact.
Even after reading the FAQs it is confusing to understand certain terms. Probably explaining them with the help of an example (something related to daily life or for a layman) would certainly improve the message of the website.
You have an excellent retention scheme a + for a marketing tool site. Add the business customers that want to share their success stories on the site, add information on the training aspect, (all BA’s are good in one thing only add a second tool and when it goes wrong it is handy to fall back on). Add some links to some business only social networking sites to improve the target audience and add something related to recent exhibitions attended or planning to attend to increase the customer base.
All in all, here are three million dollar insights (which in hindsight may look obvious)
Though it was a great experience doing usability testing, there were minor hiccups along the way. If you use a service like Feedback Army, my suggestions would be to:
I hope I have convinced you to do at least one usability test. It doesn’t cost much and is definitely worth the money. So, why don’t you try doing a usability test today?
(By the way, while writing this article, it also struck me that such usability testing is an excellent precursor to A/B testing for providing testing ideas. So, integrating Mechanical Turk within Visual Website Optimizer (an A/B split testing tool) to get feedback ahead of a split test can be a great idea. What do you think?)
Subscribe blog via RSS
