Comments on: Using A/B split testing to reduce bounce rate by 20% for an eCommerce store

By: Human Photos Double your Conversion Rate | Carsonified

Human Photos Double your Conversion Rate | Carsonified — Mon, 10 May 2010 17:01:57 +0000

[...] their first A/B test as they keep playing with their homepage. In the past, they successfully reduced bounce rate by 20% by testing the placement of their promotional [...]

By: What you really need to know about mathematics of A/B split testing « I love split testing – Visual Website Optimizer Blog

Wed, 28 Apr 2010 14:21:38 +0000

[...] you really need to know about mathematics of A/B split testing Recently, I published an A/B split testing case study where an eCommerce store reduced bounce rate by 20%. Some of the blog readers were worried about [...]

By: Rex Dixon

Rex Dixon — Tue, 16 Feb 2010 20:20:35 +0000

We would like to see you upload this test result to A/B Tests ( http://www.abtests.com/ ) – this would be a very interesting discussion for the testing community concerning conversions vs bounce rate. Great stuff here!

By: admin

admin — Fri, 08 Jan 2010 02:35:50 +0000

@Jan: the test is statistically valid. You need a lot of traffic when conversion rates are low (1% or 2%) but when conversion rates are higher (in this case 60%), a small sample size suffices the need.

Of course, I agree, higher traffic will give you more confidence.

By: admin

admin — Fri, 08 Jan 2010 02:29:35 +0000

@j-dawg: Yes, I agree that would be unwise to draw conclusions for this test (or in fact, any other A/B testing case study) and apply it directly to some other website. Every website is unique and every website needs to do a split test on its own to verify if the results are applicable.

No, MedaliaArt did not do any sort of design validation exercise prior to launching this test. What is your experience with such activities? It sure sounds like an interesting activity.

By: Jan

Jan — Thu, 07 Jan 2010 20:18:38 +0000

This study clearly showed the importance of A/B testing. The only problem I could see with the result was its statistical validity as 100 something visitors per treatment is mostly not enough.
This could backfire, because you believe the treatment outperformed the original, so you start using it. But without statistically valid data (because of not enough of visitors per treatment) you could actually hurt your conversions.
Anyway, good reading.

By: j-dawg

j-dawg — Thu, 07 Jan 2010 19:23:36 +0000

At first glance, the “Holiday Sale” design looks like some sort of paid ad – so it’s not surprising that users would bounce off a page that appears filled with annoying ads.

As many other tests have shown, offers and sales reduce bounce and increase conversion… so it’s scary to draw the wrong conclusions about offers + bounce from this test. Rather than (or in addition to) testing the messaging, a visual design/treatment test may be in order. Make it look less like an ad and more like a valuable offer.

Question: Did MedaliaArt do any sort of design validation exercise prior to launching this test? (e.g., fivesecondtest.com)