The Next Generation of Visual Website Optimizer is launching in April 2014 See What's Coming


Find Articles

Why is a winner declared earlier in the test, but then the results become even at the end?

When a new A/B split or goes live, there are multiple factors at play which result in reports that look like these:

A/B testing report

A/B testing report

Why does this happen?

Reports show so much variation in the beginning because even small changes in goals result in large percentage changes (and the default view of Visual Website Optimizer reports is the percentage view). As more data is collected, the reports also tend closer to the correct conversion rate. The screenshot below illustrates how this works.

You see winners earlier in the test because the data says so. However, you must understand that initial data is based on the actions of a few visitors and is not a true representation of the behavior of your site’s traffic. Additionally, in the real world, A/B tests reports can vary in the beginning because regular users see something new on the website and behave differently.

How to avoid making incorrect decisions

Here are some general guidelines to ensure you don’t make wrong decisions:

  1. Use our A/B test duration calculator to calculate for how long you should run your test. Even if your test is 99% significant, let it run for at least a week.
  2. If you see very large variances in the beginning, look at the Day-wise report instead of Cumulative report to get a better understanding of what’s happening:
  3. To be completely sure, wait for tests to achieve 99% significance.
VN:F [1.9.22_1171]
Rating: 3.3/5 (8 votes cast)
Why is a winner declared earlier in the test, but then the results become even at the end? , 3.3 out of 5 based on 8 ratings
No Comments

No comments yet.

Leave a comment →

Raise Support Ticket


Raise a Support Ticket!

Your Name

Your Email

Your Phone

Your Query

Or, contact us through email

A B Testing Tool