Visual Website Optimizer

Aren’t all non-converting visitors a mistrial?

i.e., if I multiple the number of visitors by 10x, but keep the conversions the same, the statistical significance of the results should not change.

See http://blog.asmartbear.com/easy-statistics-for-adwords-ab-testing-and-hamsters.html

@Portman: No, the number of visitors in the test influence the standard deviation and hence the significance. Suppose you have 10 visitors and 2 conversions v/s 1000 visitors and 200 conversions. You have a much better idea of conversion rate in the latter than the former.

[...] A/B testing significance calculator (spreadsheet in Excel) « I love split testing – Visual Websit… You just need to provide number of visitors and conversions for control and variations. The spreadsheet will automatically calculate for you significance, p-value, z-value and other relevant metrics for any kind of split testing (including Adwords). (tags: a/b testing metrics calculator spreadsheet free tools abtesting conversion) [...]

[...] a site seal, I strongly recommend performing an A-B test for a few months, or at least until some statistical significance is reached, to see if it will be worth spending the money on the seal again. Also make sure you are [...]

http://www.cliffsnotes.com/study_guide/Point-Estimates-and-Confidence-Intervals.topicArticleId-25951,articleId-25932.html

When you are using the values 1.65 and 1.96 to calculate significance isn’t that the niveau for 90% and 95 % respectively? At least, that’s what I take from the other website.

@Benjamin: you will notice that it is +/- 1.65 * SE so that covers the full 95% of area of normal curve.

Fail. Your Conversion rate limits overlap at the 95% level but you say that they are significant. This is inconsistant.

@Dennis: not sure if I got your point. Can you elaborate?

Sure, in your spreadsheet your 95% conversion rate limit for the control is between 5.68% and 7.62% while the conversion rate for the variation is between 4.81% and 6.89%. These two ranges overlap and thus you have failed to find a significant difference as the conversion rate may be 6% for the control and 6% for the variation.

However you have listed in another box that your conversion rate at 95% confidence is significant.

This result contradicts your 95& conversion rate limits results.

Could you please respond to the last comment posted by Dennis? It does seem your worksheet contradicts itself. I would like to use it, but I want to make sure it is accurate.

@Joe and @Dennis: actually, 95% range of conversion rate is different from being significant at 95% confidence level. If you visualize conversion rate ranges as a normal curves, then the overlap in 95% range constitutes a tiny area and that’s why the resultant z-value becomes significant at 95% confidence level.

I hope I am clear. If not, let me know. Will try to clarify.

Isn’t it because you are using 1.65 instead of 1.96? If you are doing a two-tailed test, 1.65 only gives you a 90% range. 1.96 is required for a 95% range, again on a two-tailed view. If you define it as checking if variation is better than control (pvariation-pcontrol<=0), then you could use a one tail range maybe. But it seems your calculator is just trying to show if they are different (i.e., you care if either one is larger than the other).

@Joe: Yes, you are right. It isn’t a one-tailed test. It depends on how you are interpreting the result but I am glad you clarified.

For the online version of the calculator, you set the minimum N as being 15. Does n=15 have an special relevance?

@Bartek: which N are you talking about?

Could I use this tool for evaluating responses to a survey?

E.g. 1000 respondents, 600 are satisfied, 400 are not satisfied. is the difference statistically significant?

[...] User There's a handy spreadsheet with statistical significance formulas on http://visualwebsiteoptimizer.co…And like Michelle Wyatt said, you can (and should) start testing with any amount of traffic, it may [...]

[...] Word of caution. Be aware of premature e-finalization. Don’t end tests before data is finalized (aka statistically significant). [...]

[...] Paras Chopra from Visual Website Optimizer has some helpful tips on figuring out when you have reached “Statistical confidence.” [...]

What’s the best way to measure statistical significance of revenue improvements. I have my split test feeding data into Analytics but I’m interested in knowing at what point my Per Visit Value (which may not correlate well with raw conversions) becomes statistically relevant. Is their a way of calculating this? To me, the answer isn’t at what point the number of conversions becomes statistically relevant it’s at what point the £ or $ becomes relevant.

@Tim: mathematically, the basis for calculating significance on revenue improvement is similar. You simply need to input the mean and standard deviation of revenue and rest of math remains the same. We already do it for revenue tracking feature in VWO: http://visualwebsiteoptimizer.com/split-testing-blog/revenue-tracking-for-ab-testing/

That looks great and would def. give me a reason to use VWO next time. I have a lot of data at the moment in GWO / Analytics for this test that we’ve run so in this particular instance I’ll need to find a way of calculating that significance with the data I’ve got.

[...] it takes for you to know which one is the best. You need statistical significance. There’s a significance calculator spreadsheet in Excel you can [...]

[...] it takes for you to know which one is the best. You need statistical significance. There’s a significance calculator spreadsheet in Excel you can [...]

The significance level of the test is not determined by the p-value, nor is it the probability that the null hypothesis is true.

One rejects the null hypothesis when the p-value is less than the significance level alpha, which is often 0.05

The p-value is based on the assumption that a result is the product of chance alone, it therefore cannot also be used to gauge the probability of that assumption being true.

The significance level of a test is a value that should be decided upon by the person interpreting the data before the data are viewed,and is compared against the p-value or any other statistic calculated after the test has been performed.

The real meaning is that the p-value is the chance of obtaining such results if the null hypothesis is true.

[...] A/B testing significance calculator (spreadsheet in Excel) Tweet This entry was posted in Uncategorized. Bookmark the permalink. ← Skal kundens navn med i emnelinjen på dit nyhedsbrev? [...]

[...] lite datagrunnlag – bruk denne enkle A/B-testing kalkulatoren for å sikre [...]

Can anybody tell me how to derivate the formula of the Z-Score? I need this formula for my thesis, so it would be good if I could explain the correctness of this formula with mathematical literature. Does anybody now books or any other scientifical papers that describe this forumla?
Thank you very much!