The Next Generation of Visual Website Optimizer is launching in April 2014 See What's Coming

Why Google Content Experiments might not fulfil your testing needs

Posted in News on August 2nd, 2012

Google Website Optimizer is morphing into the watered down Google Content Experiments, which is now available to everyone. A lot has already been written about how this will affect the Conversion Rate Optimization industry. However, resources on what this means for business owners are few and far in between. Therefore, we wrote this guide on what GWO closing down means for your online business and what steps you must take to remain ahead of the curve.

In addition to this blog post, we also have this guide in an easily digestable pdf format that you can download instantly.
Why Google Content Experiments might not fulfil your testing needs

 

1) Integration with Google Analytics – First and foremost, Content Experiments is now a part of Google Analytics and this means better integration between the two. You only need one Google Analytics UA number unlike previously, where two different UA numbers were required. This also means less code to be placed on the page because you’re inserting only one tag now. Creating experiments are still fairly easy but you’ll have to use Google Analytics goals whereas in Google Website Optimizer, you had the ability to consider any page or button a conversion goal.

You’re also limited to creating only Split URL tests, i.e. your visitors will be split between two different pages whereas other tools offer A/B testing on the same page (visitors are split between different versions of the same page).

2) Your experiments will be terminated after 3 months – All content experiments currently have a maximum life of 3 months. If you want to see data after that, you’ll have to create a new experiment and combine the data between the two. Usually, small improvements take a lot of time to achieve statistical significance while large improvements happen fairly quickly. If any of your experiments used to take longer than 3 months, you’ll have to learn to create bigger changes that show bigger results.

3) Experiments have to run for 2 weeks before Content Experiments reports statistical significance – This is a constraint if you have a large number of visitors to your website and statistical confidence can be calculated in a timeframe lesser than 2 weeks. In such a scenario, you could end up losing sales as part of your visitors will be shown a low performing variation for a period of time that is longer than required. However, test results will be shown live.

4) Advanced Segments will be available to slice & dice your data – All Advance Segments are available, meaning that you can now very finely slice & dice your data and draw better insights about your visitors’ behavior. This is a very useful addition to GCE that will enable you to finetune your funnel for different customer segments.

5) You cannot stop low performing variations before the test is over – In a running test, you will sometimes see a variation that is obviously not doing well and turn it off. This is especially so in large eCommerce websites where rogue unstoppable variations could result in a significant loss of revenues. While GCE won’t have a way to stop low performing variations during a running test, it will try to dynamically shift traffic away from them.

6) You can have maximum 5 variations per test and not more than 12 active tests – These shouldn’t be a hindrance for most small businesses but the same cannot be said for larger eCommerce websites. Websites with a lot of traffic benefit greatly from regular testing on different steps of the funnel and limiting the number of tests will prove to be a pain point.

7) You can now use URL variables and Regular Expressions – You can run tests across domains and include / exclude URLs using regular expressions. This will really help to match the tests to a path as your visitor will move from page to page. REGEX and URL variables are also helpful when you want to exclude certain IPs from a test, for example, your own company IPs so that your actions do not count as goals and skew the results.

8) No multivariate testing – This is a very important feature that Google inexplicably decided to leave out. Multivariate testing enables a quicker understanding of which changes work and which don’t. The absence of this feature is a considerable drawback for large eCommerce websites who have the visitor flow to achieve statistical significance quickly.

9) Email notifications are now available – Useful feature because you won’t have to monitor your tests constantly – set up the notifications and let Google tell you what’s happening when.

10) Revenue goals are missing – Another important functionality that Google should have included in GCE is the ability to track eCommerce transactions as goals. Currently, only URL and events based goals can be tracked so you’ll have to use a thank-you page to achieve the desired outcome. It’s not a crippling feature, but is still a “hack”.

11) Different code snippets for different experiments – What they should be aiming for is one single code snippet that allows you to run multiple tests on your page. For example, Visual Website Optimizer requires you to place the code snippet only once on a page and then, any number of tests can be run on the page simultaneously. For GCE, each Experiment has its own code snippet and you can have only one running at a time. This has two implications

  • You can’t run more than one test on a step of the funnel meaning it takes longer for you to optimize your website and see the result;
  • If you want to run a second test after 2 weeks, you’ll have to dig in and replace the code snippet.

In conclusion, Google Content Experiments is a tool most suitable for those just starting out with A/B testing and conversion rate optimization. It has all that any beginner would require to understand how CRO works and what it can do for an online business.

However, if you test regularly and have a website with a large number of daily or monthly visitors, you’d be well advised to go for a full fledged A/B split and Multivariate testing alternative like Visual Website Optimizer. GCE is still evolving and we are sure newer features will be rolled out in the coming days as the team behind it collects user feedback, but until then your business will need a more robust package that handles all your testing needs effortlessly.

Siddharth Deswal

I do marketing at Visual Website Optimizer.

The Complete A/B Testing Guide

Know all that you need to get started:

  • What is A/B Testing?
  • Is it compatible with SEO?
  • How to start your first A/B test?
Show Me The Guide


Tags

5 Comments
Marty Foley
August 2, 2012

I considered many of the factors you mention in your post, before making the switch from GWO to VWO.

toni
August 6, 2012

we are very disappointed about the current situation with google experiments.
We have a lot problems with the integration.
First it doesn’t show any data, now it show us transcation but no visitors.
The help is very poor and the support terrible. I don’t understand why tests on the same site are not possible!
Dear Google, its not suffice just to be a good search engine, also the other products should work “normal”

Aurelien
August 9, 2012

I’ve tested Google Content Experiments with a sales page and the variation was getting far less visitors than the control, plus the experiment stopped after about one and a half month, leaving me with an uncertain result.

Pretty disappointing, I don’t understand why they got rid of GWO that was a great tool…

Sean Mize
August 23, 2012

Yeah, GWO closing down was one of the biggest reasons I started looking for an optimization service, then I found you guys – I have been amazed at how EASY VWO is to use! I just create experiments in minutes on the fly now, it used to take so long to set up one in GWO that I would often run without optimization.

So thanks, VWO!

Sean

Daniel
August 30, 2012

I actually find Split Testing very interesting as I have read it as well from another article regarding split testing tools and I see that it will be very useful to drive companies marketability and sales tactics. I will check out GCE much further and will see if I can have testers work it out for our marketing.

Sorry, the comment form is closed at this time. To start a discussion, please email info@wingify.com.

RSS feed for comments on this post. TrackBack URL

I ♥ Split Testing Blog


Stay up to date

Recent Posts

Write for this blog!

We accept high quality articles related to A/B testing and online marketing. Send your proposals to contribute@wingify.com