Editor’s note: this is a case study written by one of our customers (InvisibleHand). Usually A/B testing is limited to web pages but they did an interesting A/B test which actually worked inside a browser extension. It’s one of a kind A/B testing case study and we are thankful to them for sharing it. So, go ahead and enjoy the case study!
Most people refer to web pages when they talk about A/B testing. However, A/B testing can be useful in other domains too. At InvisibleHand, we use Visual Website Optimiser to test the performance of different versions of a browser extension.
InvisibleHand is a browser extension that automatically compares prices for you. For example, when you browse a product at Buy.com, InvisibleHand checks other retailers where this product is available, compares real-time prices and shows a notification like “Save 10%! This product is $10 cheaper ($89.99) at Amazon.com”, giving you a direct link to the product at a competing retailer. Since the launch in 2009 InvisibleHand enjoyed almost 2m downloads. We make money by earning commission on affiliate sales, so our revenues depend on the quality and quantity of our notifications.
In order to make InvisibleHand available on all major browser platforms (Chrome, Firefox, Safari, IE), we try to reuse as much code as possible. In particular, the notification shown is injected as an iframe (as opposed to a div element, for example, or the built-in browser API) to enable us to share the same code across browsers and to control the behaviour of the extension from the server-side without having to ask users to install a new version every time we try a new feature. Crucially, this also enables us to perform A/B testing of new functionality.
Normal notifications in InvisibleHand
The iframe has significant control over the look of the notification. In particular, it can choose whether to be shown, which comes really handy if we decide to enable or disable certain kinds of notifications. The mechanism is simple:
Since the iframe is just a web-page, we can use Visual Website Optimiser to tests different variations of the extension. For example, the very first test we performed measured the likelihood of a click on a notification using two different fonts. However, what we really are interested in is what makes our users to uninstall InvisibleHand and how to prevent it.
Our first hypothesis was that some of our notifications are more annoying than the others and they prompt the users to remove the extensions. In particular, we show money-saving notifications on google and other search engines when product searches are performed but these notifications are not highly accurate since the users are not always searching for a product and even when they are, they may not be interested in buying it. For example, a user may be researching a camera online and InvisibleHand notifications would be just a distraction if they don’t plan on buying it right now.
Annoying search notifications in InvisibleHand
So, we designed a variation of the test that doesn’t show notifications on search engines. In our iframe we have a check:
var overloading = true;
if (typeof shouldShowNotification == 'function') {
overloading = shouldShowNotification(this);
}
if (eventSource && overloading) {
eventSource.postMessage(JSON.stringify({message: 'showNotification'}), '*');
}
This piece of code shows all notifications by default unless there is a function shouldShowNotification() defined that can keep a notification hidden. So we created a variation for A/B testing that defines this function:
function shouldShowNotification(notification) {
return notification.category != ‘search_engine’;
}
In this variation all notifications on all search engines would remain hidden even though they would still be injected into the page.
I mentioned earlier that InvisibleHand works with all major browsers but we usually introduce experimental features to only one platform, Chrome, to avoid the hassle of deployment and rollbacks. So, to limit the scope of the test to only one browser, we check the user agent:
var browserEnabled = navigator.userAgent.toLowerCase().indexOf('chrome') > -1;
if (browserEnabled) {
var _vis_opt_account_id = 7013;
//... the rest of the code
So, the VWO code isn’t executed on all other browsers. Cool!
Measuring extension uninstallations on Google Chrome turned out to be slightly trickier. Since the uninstallation of a browser extension may happen when no notifications are shown and VWO code is not running, we decided to show a “goodbye” notification when the extension was uninstalled. The “goodbye” notification is hosted on the same domain as the normal notification; its primary purpose is to trigger a custom conversion goal:
_vis_opt_register_conversion(1, 5);
It’s ironic that Visual Website Optimiser assumes that we want to maximise all conversion goals, so when the uninstallation rate was going up it was reported as an “improvement” [Editor's note: great point! We will work on it]. Other than that, it worked just as expected, allowing us to measure the correlation between annoying notifications being disabled and the probability of uninstallation. The conclusion was that there is indeed a slight increase in the number of uninstallations when the notifications on search engines are shown but since the engagement rate (number of clicks) on these notification is fairly high, we concluded that the users would probably be better off with them rather than without.
One of the reasons we decided to move part of the functionality of InvisibleHand into an iframe is the ability to perform A/B tests and we’re quite happy with Visual Web Optimizer helping us to do this. As it turns out, it’s perfectly possible to A/B test browser extensions, as long as some parts of them are normal web pages.
Squeeze pages are special type of landing pages which try to collect email address of a visitor by promising a free offer. Squeeze pages usually have high conversion rate (typically >10%) since the offer is compelling and call to action is clear. Here’s how a typical squeeze page looks like:
Squeeze page example
Note the provocative headline, prominent call to action and a form to collect email address. That’s all this page has but it works wonderfully!
Rishan (from 4D Innovations) is one the Visual Website Optimizer customers and he runs a business that empowers and guides people to achieve financial freedom by doing what they love in the context of the online marketplace. He decided to do a split test between three squeeze pages (with same offer but different content) and amazingly he found a version that converted at an impressive 28%. That means for 100 visitors, he was able to persuade 28 visitors to give their email address to them in exchange of free guide. This conversion rate was an improvement of 125% of his original page (which had 12% conversion rate).
Just by changing the design of the squeeze page, Rishan more than doubled his conversion rate. No new offers, same traffic but now 125% more email addresses collected for the list. How cool is that?
So, till now I have just talked about the split test. But where is the screenshot of the highest converting squeeze page?
Well, following are different versions tested in the split test and their corresponding conversion rates:
Original page: 12.5% conversion rate
Variation B: 21% conversion rate
Best performing design (Variation A): 28% conversion rate
Variation A converted at 28% and improved conversions by 125% over the original version (the difference was statistically significant).
Interesting point to note is that Rishan never expected variations to perform better as control had the opt-in form above the fold and winning variation didn’t. That’s why he said: “There’s no way to find out which design converts best unless you do an A/B test!”
Starting with a new series on this blog called Revisiting the Classics, I will re-look at our A/B testing case studies and throw some new light on it (with additional commentary). The case study that I picked for this post relates to human photos on landing pages and whether they increase conversion rate. Before reading rest of the post, make a guess: do human photos (male/female) online really have any subconscious effect on visitors?
There has been significant scientific research on this topic particularly for eCommerce websites. When you are considering a purchase decision, the main issue online is of trust. How do you know if a particular website is trust worthy? Taking lessons from TV advertising and general marketing principles, people running eCommerce websites think that associating photos of people with products engenders trust. Of course, human photos = increased trust is just a hypothesis. But is it really true?
To answer this, I analyzed several research papers (which are published in peer-reviewed scientific journals). Following are some of the main ones (along with excerpts):
Not just scientific research, real world A/B tests that I have seen also corroborate the hypothesis of human photos may increase conversion rate. Two of our customers used our A/B testing tool (Visual Website Optimizer) to test presence of human photos and its impact on conversions. Following sections detail what they did and what they found out.
Medalia Art sells Brazilian and Caribbean art online. Since they are an online art shop, they showcase paintings from famous artists on their homepage. They decided to test what happens if paintings are replaced by the artists’ photos.
The goal for this A/B test was to increase visitor engagement (defined as a click on any link on the homepage; it is the inverse of the bounce rate). They didn’t use sales as a conversion rate as many of their sales happen after interaction on the telephone with the customer.
Here are the screenshots of the control (paintings) and the winning version (photos).
- Original (with paintings)-
- Variation (with photos)-
Results?: Medalia found out that with paintings the conversion rate was 8.8% but if paintings are replaced with artist photos, the conversion rate increases to whopping 17.2%. That’s an increase in conversion rate of more than 95%! (results were statistically significant) How cool is that for doing a slight change?
Another user, Jason Thompson, conducted an A/B test on his blog to see if replacing contact icon with his own photo would lead to more people contacting him. Following is screenshot of original and variation:
|
Original (with icon) |
Variation (with photo) |
 |
 |
|
3.7% conversion rate |
5.5% conversion rate |
Results clearly show that the version with Jason’s photo had 48% more conversions as compared to generic icon. (Again, results are statistically significant).
Jason commented on the test results:
People want to connect with other people emotionally, the photo makes that emotional connection so much easier and as the test is proving, drives people to the contact form more than a nondescript icon.
There are hundreds of more research studies and probably more A/B test results that you can read but from what I have read and observed, here are some salient points about human photos on websites:
Hope you found this article useful. If you have any comments or suggestions, please let me know.
Beamax, a Belgium based company, manufactures and distributes projection screens for home cinemas and meeting rooms world-wide. They wanted visitors on homepage to go to a site dedicated to ex-demo and one-off items that are sold directly to consumers. (All other screens are sold indirectly through resellers). They admit that it’s a bit odd to drive away visitors to another site from your main page, but they wanted to clear up some space in the warehouse that was taken up by uncommon items.
So, they decided to do a simple A/B test using Visual Website Optimizer. Just above the product images on homepage, they put a standard link promoting the other website. It said:
Great deals on brand new and ex-demo screens here
To increase clickthroughs on link, they tested a red link (with same text) because they felt it would out-perform the standard blue that they use. Plus, it’s something direct marketers use in “real” mail pieces too. As another variation, they transformed link into something banner-like that they thought would even have more impact. Their hypothesis was that the banner version the sure-fire winner. See screenshots below:
Original page
Page with red link
Page with red banner
So, any guesses on which version got maximum clicks: blue, red or banner version? Well, the red link and banner both outperformed the blue link and that wasn’t a surprise. But the eye-opening result was that the red link winning from the banner. The improvement of red link compared to the original blue link was pretty big too: 53.13%. (Note we have some other case studies online which demonstrate how red link outperforms the default link. Here are two examples: PDFProducer case study and GSM.nl case study)
Otto Tromm, CEO of Beamax, stresses the importance of waiting for statistically significant results. He says:
In the early stage of the test, the banner was the big winner. But, over time (when the results got more reliable), the red link outperformed the banner. That taught me not to jump to conclusions.
And it was tempting to declare an early winner, because initial results proved my gut feeling. The test proved me wrong, so it teaches you to stay humble too.
So would I implement a red link vs a banner blindly next time? No, I would test it!
Choice of the right tool is certainly very important when are you are doing A/B tests. Beamax chose Visual Website Optimizer for the job (just like thousands of other businesses). Here’s what Otto from Beamax has to say about the tool:
I am not a designer or coder and we use Mod-x and CMS defined templates for nearly all pages. So I neither want to call on experts for every test I do, nor do I want to mess up their work. Visual Website Optimizer made it easy for a non-tech guy to do the tests and keep our designer and programmer focused on their own projects.
With Google’s solution, it was a lot more work to implement tests, which is why I stopped using it. Just couldn’t get it all done myself, which is important when you have an idea and quickly want it implemented.
Hope you liked this case study! If you have any comments or suggestions, we are all ears.
Red Gate software runs an annual challenge where they buy a small software company for a million dollars. They list a number of requirements that the software company must fulfill. One of the requirements that stood out was about conversion rate. They said:
If you’re selling your product then it must have at least a 10% conversion rate.
This requirement actually made me say “Wow, that’s insane”. Let me elaborate why.
Conversion rate is percentage of visitors who actually bought something on your website. Let’s imagine there are two websites: one sells product X with 5% conversion rate and the other one sells product Y with 10% conversion rate. Now, here is a million dollar question:
Is 10% conversion rate of product Y better than 5% conversion rate of product X?
It’s foolish to even begin answering above question without considering following factors:
In nutshell, conversion rate by itself doesn’t tell much (unless you have extra information like traffic, sales price, lifetime value, traffic mix, etc.) So a website with 1% conversion rate may not necessarily be worse as compared to a website with 10% conversion rate. Conversion rate in isolation is a useless metric.
Conversion rates are not entirely useless. In fact, they are very useful when seen on a temporal scale. In other words:
If your conversion rate is 5% today, aim should be to increase it to 7% (using A/B testing, etc,) or at least not let it fall to 3%.
So, comparing conversion rate over time makes a lot of sense (but for the same website). Unless you have a lot of other information about your competitors, you should NOT obsess over comparing your conversion rate to their conversion rate and whether it is lower/higher. Instead, you should obsess on how you can increase your conversion rate (since that’s one of the easiest things to make your bank balance fatter).
Note: if you go through our library of A/B testing case studies, you will note that we always talk about increase in conversion rate and not conversion rate per se.
