The Next Generation of Visual Website Optimizer is launching in April 2014 See What's Coming

A/B testing browser extensions: do annoying notifications cause uninstallations?

Posted in A/B Split Testing, Case Studies on May 31st, 2011

Editor’s note: this is a case study written by one of our customers (InvisibleHand). Usually A/B testing is limited to web pages but they did an interesting A/B test which actually worked inside a browser extension. It’s one of a kind A/B testing case study and we are thankful to them for sharing it. So, go ahead and enjoy the case study!

Most people refer to web pages when they talk about A/B testing. However, A/B testing can be useful in other domains too. At InvisibleHand, we use Visual Website Optimiser to test the performance of different versions of a browser extension.

InvisibleHand is a browser extension that automatically compares prices for you. For example, when you browse a product at Buy.com, InvisibleHand checks other retailers where this product is available, compares real-time prices and shows a notification like “Save 10%! This product is $10 cheaper ($89.99) at Amazon.com”, giving you a direct link to the product at a competing retailer. Since the launch in 2009 InvisibleHand enjoyed almost 2m downloads. We make money by earning commission on affiliate sales, so our revenues depend on the quality and quantity of our notifications.

A/B testing inside browser extension using IFrames

In order to make InvisibleHand available on all major browser platforms (Chrome, Firefox, Safari, IE), we try to reuse as much code as possible. In particular, the notification shown is injected as an iframe (as opposed to a div element, for example, or the built-in browser API) to enable us to share the same code across browsers and to control the behaviour of the extension from the server-side without having to ask users to install a new version every time we try a new feature. Crucially, this also enables us to perform A/B testing of new functionality.

Normal notifications in InvisibleHand

The iframe has significant control over the look of the notification. In particular, it can choose whether to be shown, which comes really handy if we decide to enable or disable certain kinds of notifications. The mechanism is simple:

  • The extension injects an invisible empty notification and sends a message with notification contents using postMessage()
  • The iframe analyses its contents and decides what to do
  • The iframe sends a message back to the notification telling it to make the iframe visible if necessary

Since the iframe is just a web-page, we can use Visual Website Optimiser to tests different variations of the extension. For example, the very first test we performed measured the likelihood of a click on a notification using two different fonts. However, what we really are interested in is what makes our users to uninstall InvisibleHand and how to prevent it.

Hypothesis: do annoying notifications cause uninstallations?

Our first hypothesis was that some of our notifications are more annoying than the others and they prompt the users to remove the extensions. In particular, we show money-saving notifications on google and other search engines when product searches are performed but these notifications are not highly accurate since the users are not always searching for a product and even when they are, they may not be interested in buying it. For example, a user may be researching a camera online and InvisibleHand notifications would be just a distraction if they don’t plan on buying it right now.

Annoying search notifications in InvisibleHand

A/B test implementation

So, we designed a variation of the test that doesn’t show notifications on search engines. In our iframe we have a check:

    var overloading = true;
    if (typeof shouldShowNotification == 'function') {
      overloading = shouldShowNotification(this);
    }
    if (eventSource && overloading) {
      eventSource.postMessage(JSON.stringify({message: 'showNotification'}), '*');
    }

This piece of code shows all notifications by default unless there is a function shouldShowNotification() defined that can keep a notification hidden. So we created a variation for A/B testing that defines this function:

   function shouldShowNotification(notification) {
      return notification.category != ‘search_engine’;
   }

In this variation all notifications on all search engines would remain hidden even though they would still be injected into the page.

I mentioned earlier that InvisibleHand works with all major browsers but we usually introduce experimental features to only one platform, Chrome, to avoid the hassle of deployment and rollbacks. So, to limit the scope of the test to only one browser, we check the user agent:

    var browserEnabled = navigator.userAgent.toLowerCase().indexOf('chrome') > -1;
    if (browserEnabled) {
      var _vis_opt_account_id = 7013;
      //... the rest of the code

So, the VWO code isn’t executed on all other browsers. Cool!

Measuring uninstallations!

Measuring extension uninstallations on Google Chrome turned out to be slightly trickier. Since the uninstallation of a browser extension may happen when no notifications are shown and VWO code is not running, we decided to show a “goodbye” notification when the extension was uninstalled. The “goodbye” notification is hosted on the same domain as the normal notification; its primary purpose is to trigger a custom conversion goal:

    _vis_opt_register_conversion(1, 5);

A/B test results

It’s ironic that Visual Website Optimiser assumes that we want to maximise all conversion goals, so when the uninstallation rate was going up it was reported as an “improvement” [Editor's note: great point! We will work on it]. Other than that, it worked just as expected, allowing us to measure the correlation between annoying notifications being disabled and the probability of uninstallation. The conclusion was that there is indeed a slight increase in the number of uninstallations when the notifications on search engines are shown but since the engagement rate (number of clicks) on these notification is fairly high, we concluded that the users would probably be better off with them rather than without.

One of the reasons we decided to move part of the functionality of InvisibleHand into an iframe is the ability to perform A/B tests and we’re quite happy with Visual Web Optimizer helping us to do this. As it turns out, it’s perfectly possible to A/B test browser extensions, as long as some parts of them are normal web pages.

Paras Chopra

CEO and Founder of Wingify by the day, startups, marketing and analytics enthusiast by the afternoon, and a nihilist philosopher/writer by the evening!

The Complete A/B Testing Guide

Know all that you need to get started:

  • What is A/B Testing?
  • Is it compatible with SEO?
  • How to start your first A/B test?
Show Me The Guide


Tags

5 Comments

Wow, that’s an incredibly creative way to do split-testing. I’d personally have no use for doing that but kudos to you guys for getting the idea and figuring out how to implement it.

Great stuff!

Jeremy Reeves

[...] colleague – Evgeny – has guest-written a great post on the use of A/B testing on [...]

[...] in-depth description of how we split test InvisibleHand, our price comparison extension. LikeBe the first to like this [...]

How to Design a Website
August 4, 2011

A/B testing, split testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.

Significant improvements can be seen through testing elements like copy text, layouts, images and colors. However, not all elements produce the same improvements, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest improvements.

Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. The test, in order to be effective, must reach an audience of a sufficient size that there is a reasonable chance of detecting a meaningful difference between the control and other tactics: see Statistical power.

——————–
Stephen

[...] 10. A/B testing browser extensions: do annoying notifications cause uninstallations? [...]

Leave a comment
Required
Required - not published


5 − four =

Notify me of followup comments via e-mail.

RSS feed for comments on this post. TrackBack URL

I ♥ Split Testing Blog


Stay up to date

Recent Posts

Write for this blog!

We accept high quality articles related to A/B testing and online marketing. Send your proposals to contribute@wingify.com