Comments on: What you really need to know about mathematics of A/B split testing

By: A/B spilttest giver ikke et brugbart resultat

A/B spilttest giver ikke et brugbart resultat — Sat, 31 Dec 2011 12:26:57 +0000

[...] læsning What you really need to know about mathematics of A/B split testing Tweet This entry was posted in Uncategorized. Bookmark the permalink. ← Skal kundens [...]

By: Paras Chopra

Paras Chopra — Tue, 13 Dec 2011 12:29:32 +0000

@Rafael: yes, your understanding is correct.

By: Rafael

Rafael — Tue, 13 Dec 2011 12:15:38 +0000

Hey Paras, thank you for the answer!

So the “chance to beat the control” can be measured just by measuring this overlap?

For instance, does a 10% overlap mean a 90% chance to beat the control?

And a 15% overlap -> 85% chance
20% overlap -> 80% chance

And so on, so forth. Is this the case, or did I misinterpret it?

By: Paras Chopra

Paras Chopra — Tue, 13 Dec 2011 07:46:27 +0000

@Rafael: it depends on how important results are for an organization and how much risk (of being wrong) it is willing to accept. 99% chance to beat original is always better, but if stakes aren’t high some organizations are okay with 95% chance to beat original too.

By: Rafael

Rafael — Mon, 12 Dec 2011 22:57:03 +0000

How much overlap is allowed between the two distributions to be confident that version B is better?

You said that if there is 1% overlap between conversion rate distribution of control and variation, then there is 99% chance of variation beating the control.

What if there is a 5% overlap? In this case, is there a 95% chance of the variation beating the original?

What about a 6% overlap?

Thanks!

By: Paras Chopra

Paras Chopra — Mon, 12 Dec 2011 08:28:06 +0000

@Andi: it is a binomial distribution and for binomial distribution, variance is p * (1 – p)

By: Andi

Andi — Mon, 12 Dec 2011 08:21:02 +0000

Can you explain how you get to this formula, please?
Standard Error (SE) = Square root of (p * (1-p) / n)

I don’t understand how you can calculate the standard error without knowing anything about the variance.
That would be really helpful, thank you!

By: Dossier metrics part4, pour aller plus loin: sources et ouvrages | Clement vouillon

Mon, 31 Oct 2011 13:58:13 +0000

[...] What you really need to know about mathematics of A/B split testing [...]

By: Introducing frequently used goals, test thresholds, custom currency and more!

Introducing frequently used goals, test thresholds, custom currency and more! — Sat, 08 Oct 2011 22:13:39 +0000

[...] by default, we declare winning variation if the (statistical) confidence is >95% (here’s the math of A/B testing if you are interested). Now from settings, you can change it to any value you want. So if you want [...]

By: Paras Chopra

Paras Chopra — Tue, 23 Aug 2011 14:28:59 +0000

@Anne: you make a good point and it will be great to capture trending into a mathematical number. However, ‘chance to beat original’ or ’statistical significance’ talks about results in overall context. With these metrics we want to understand what is the likelihood that variation is performing better as compared to control given a specific sample (over a number of days).

What you are asking is a number that says how consistent is the performance. Those are two different things but nevertheless consistency can be important too.