There are tons of A/B testing articles on the Internet. In fact, Visual Website Optimizer blog is full of them. But let’s admit it, all those guides, tips, tricks ultimately start sounding similar to each other. I decided to write a post on tips that do not have a straightforward logic and are not popularly discussed on the Internet. In fact, people usually do the opposite of what I am going to discuss in this post.

Don’t announce to the world that you are doing an A/B test

I won’t paste links here but a quick search on Twitter will confirm that people love raving about the A/B test they are currently running. You should absolutely not do this. Apart from having no real benefit, you can actually skew your results big time. With an A/B test, you want to determine how well variations work on your usual website traffic. Normal site visitors and curious visitors looking for your new test are very different in their behavior – you don’t want to bias your results with non-conversions of all visitors whose motive is to actually see what is being A/B tested.

Don’t launch your A/B test if you are expecting some heavy publicity

If you are expecting a spike in traffic, it is best to avoid testing because: a) majority of visitors in the traffic spike will not be like your typical website visitors so they are usually going to bias your results in a big way. Imagine for a moment that you are observing a 20% increase in conversions with variation A as compared to control. Now TechCrunch covers you and sends you bootloads of traffic. This traffic is known to be come-and-go type. So, your overall conversions will go down. Agreed that conversions will go down for all variations, however the difference between performance of variations will narrow down. If variation A was beating control by 20% earlier, after the traffic spike it will be down to 10% or so for no good reason. So, consider pausing your tests if you expect some major publicity.

Optimize for average conversion rate even while the test is running

A/B testing purists may disagree that you should set the test once and then let it complete, but I say you should definitely keep an eye while the test is running. Of course, the primary aim of the test is to ultimately increasing site conversion rate. But you don’t want to wait till the end and realize that you have lost many sales during the testing period. Consider disabling poorly performing variations and making variants of best performing ones in order to increase conversion rate even the test is running.

Those were 3 A/B testing tips. If you have any controversial A/B testing tip to share, do leave a comment here.

---

Tweet

Follow @wingify

Tags