One of the Visual Website Optimizer’s earliest beta testers, MedaliaArt is an online art gallery specializing in Caribbean and Latin America art. For the holiday season, they put up a sale where they give 5-55% discounts on all paintings. They wanted to determine the best location on homepage to put up that message so as to optimize for bounce rate. Their sales process is long (involving phone calls, multiple visits, etc.) so they chose to measure and optimize bounce rate instead of sales conversions. As a hypothesis, providing discounts must pull in more visitors to go through multiple pages on the website exploring different paintings.
However the challenge with putting up a ‘Holiday Sale’ message is where to show it. Displaying it prominently on the homepage will make more visitors notice it but some may find it too intrusive and leave the site immediately. On the other hand, putting it at a not-so-noticeable location may have no effect at all. So, what is the best position on page to display the ‘Holiday Sale’ (or for that matter any other promotional) message?
Only a split test can answer that. (In a split test, different visitors see (randomly selected) different versions of homepage). MedaliaArt setup a split test to optimize their website for bounce rate. First, they created a couple of versions of the homepage with ‘Holiday Sale’ displayed at different locations. Of all versions, following represented two extremes:
In-your-face ‘Holiday Sale’ message displayed in big, red font prominently on the homepage.
Sidebar ‘Holiday Sale’ message in small font.
Usually, split testing tools do not track bounce rate; they rather track conversion rate (percentage of visitors doing desired action). To track bounce rate instead, MedaliaArt did a neat trick. They defined a click on any link on the homepage as conversion. Thus the conversion rate of, for example, 40% corresponded to 100-40 = 60% bounce rate.
So, which variation had a better bounce rate? Any guesses?
They started the test and after two weeks got their first batch of conclusive results.
| Message location | Visitors | Clicks (conversions) | Conversion Rate | Bounce Rate | Reduction |
| Sidebar | 145 | 35 | 24% | 76% | N.A. |
| In-your-face | 123 | 49 | 40% | 60% | -21% |
Clearly, the in-your-face, prominent promotional message has dramatically less bounce rate (60%) than the sidebar one (76%). The reduction in bounce rate of 21% is statistically significant (at 95% confidence level) so the In-your-face variation really represents a better version. The improvement in bounce rate means more interest by visitors in the paintings they are selling and potentially more sales. What they feared that a prominently displayed promotional message can backlash by irritating visitors didn’t really happen. Without split testing they could have never really known the optimal position of their promotional message. Now they know.
For the next test they do, there are a couple of suggestions for MedaliaArt (or any other eCommerce optimizing for promotional messages):
What eCommerce stores learn from this case study?
Split testing is the only way the really know what will work and what won’t. Testing is essential to check assumptions related to promotional messages, checkout process, product category ordering, buy now button, etc. Be a little adventurous and test radically different homepage designs and ideas. You can always choose to include only a small percentage of traffic and can disable non-performing variations at a click of a button. So, what’s your excuse for not using split testing for increasing sales for your eCommerce store?
This case study is also appeared in form of interview at Practical eCommerce magazine as Split Testing Can Increase Conversion Rates.
RSS feed for comments on this post. TrackBack URL
At first glance, the “Holiday Sale” design looks like some sort of paid ad – so it’s not surprising that users would bounce off a page that appears filled with annoying ads.
As many other tests have shown, offers and sales reduce bounce and increase conversion… so it’s scary to draw the wrong conclusions about offers + bounce from this test. Rather than (or in addition to) testing the messaging, a visual design/treatment test may be in order. Make it look less like an ad and more like a valuable offer.
Question: Did MedaliaArt do any sort of design validation exercise prior to launching this test? (e.g., fivesecondtest.com)
Comment by j-dawg — January 7, 2010 @ 7:23 pm
This study clearly showed the importance of A/B testing. The only problem I could see with the result was its statistical validity as 100 something visitors per treatment is mostly not enough.
This could backfire, because you believe the treatment outperformed the original, so you start using it. But without statistically valid data (because of not enough of visitors per treatment) you could actually hurt your conversions.
Anyway, good reading.
Comment by Jan — January 7, 2010 @ 8:18 pm
@j-dawg: Yes, I agree that would be unwise to draw conclusions for this test (or in fact, any other A/B testing case study) and apply it directly to some other website. Every website is unique and every website needs to do a split test on its own to verify if the results are applicable.
No, MedaliaArt did not do any sort of design validation exercise prior to launching this test. What is your experience with such activities? It sure sounds like an interesting activity.
Comment by admin — January 8, 2010 @ 2:29 am
@Jan: the test is statistically valid. You need a lot of traffic when conversion rates are low (1% or 2%) but when conversion rates are higher (in this case 60%), a small sample size suffices the need.
Of course, I agree, higher traffic will give you more confidence.
Comment by admin — January 8, 2010 @ 2:35 am
We would like to see you upload this test result to A/B Tests ( http://www.abtests.com/ ) – this would be a very interesting discussion for the testing community concerning conversions vs bounce rate. Great stuff here!
Comment by Rex Dixon — February 17, 2010 @ 1:50 am
[...] you really need to know about mathematics of A/B split testing Recently, I published an A/B split testing case study where an eCommerce store reduced bounce rate by 20%. Some of the blog readers were worried about [...]
Pingback by What you really need to know about mathematics of A/B split testing « I love split testing – Visual Website Optimizer Blog — April 28, 2010 @ 7:51 pm
[...] their first A/B test as they keep playing with their homepage. In the past, they successfully reduced bounce rate by 20% by testing the placement of their promotional [...]
Pingback by Human Photos Double your Conversion Rate | Carsonified — May 10, 2010 @ 10:31 pm