Comments on: What you really need to know about mathematics of A/B split testing

By: 5 most common A/B testing misconceptions

5 most common A/B testing misconceptions — Thu, 19 Apr 2012 08:28:07 +0000

[...] control) is statistically significant. You can read about mathematics of it in previous blog posts: how we calculate statistical significance and how to estimate number of visitors needed for a test. A common misconception is that a winning [...]

By: Komposit

Komposit — Wed, 11 Apr 2012 12:23:16 +0000

Thanks a lot, this was just what I was looking for!

Keep up the good work.

By: La guida per condurre un Test A/B pubblicata da Smashing Magazine.

La guida per condurre un Test A/B pubblicata da Smashing Magazine. — Sat, 18 Feb 2012 15:28:51 +0000

[...] What You Should Know About the Mathematics of A/B Testing From my own blog. [...]

By: A/B spilttest giver ikke et brugbart resultat

A/B spilttest giver ikke et brugbart resultat — Sat, 31 Dec 2011 12:26:57 +0000

[...] læsning What you really need to know about mathematics of A/B split testing Tweet This entry was posted in Uncategorized. Bookmark the permalink. ← Skal kundens [...]

By: Paras Chopra

Paras Chopra — Tue, 13 Dec 2011 12:29:32 +0000

@Rafael: yes, your understanding is correct.

By: Rafael

Rafael — Tue, 13 Dec 2011 12:15:38 +0000

Hey Paras, thank you for the answer!

So the “chance to beat the control” can be measured just by measuring this overlap?

For instance, does a 10% overlap mean a 90% chance to beat the control?

And a 15% overlap -> 85% chance
20% overlap -> 80% chance

And so on, so forth. Is this the case, or did I misinterpret it?

By: Paras Chopra

Paras Chopra — Tue, 13 Dec 2011 07:46:27 +0000

@Rafael: it depends on how important results are for an organization and how much risk (of being wrong) it is willing to accept. 99% chance to beat original is always better, but if stakes aren’t high some organizations are okay with 95% chance to beat original too.

By: Rafael

Rafael — Mon, 12 Dec 2011 22:57:03 +0000

How much overlap is allowed between the two distributions to be confident that version B is better?

You said that if there is 1% overlap between conversion rate distribution of control and variation, then there is 99% chance of variation beating the control.

What if there is a 5% overlap? In this case, is there a 95% chance of the variation beating the original?

What about a 6% overlap?

Thanks!

By: Paras Chopra

Paras Chopra — Mon, 12 Dec 2011 08:28:06 +0000

@Andi: it is a binomial distribution and for binomial distribution, variance is p * (1 – p)

By: Andi

Andi — Mon, 12 Dec 2011 08:21:02 +0000

Can you explain how you get to this formula, please?
Standard Error (SE) = Square root of (p * (1-p) / n)

I don’t understand how you can calculate the standard error without knowing anything about the variance.
That would be really helpful, thank you!