Imagine that you are selling cookies online and want to increase sales. What would you do next? One way to increase sales would be to increase traffic that you get but that costs a lot of money. Another way would be to work on optimizing landing pages, product pages and checkout pages. You may test your button colors, call to actions or layout. If you are adventurous, you may move beyond simple changes and start testing entirely different offers or value propositions. Who knows if $5 discount increases sales or probably next-day shipping is more important to convince a customer to gift a box of cookies?
The client: online cookie store
These were exactly the kind of questions that (our certified partner) Trinity Insight wanted to answer for their client SmileyCookie. SmileyCookie is a niche eCommerce store that sells cookies online. They allow you to gift customized cookies to anyone for any occasion (weddings, sports events, corporate events, etc.) What could be a better gift than a box of customized cookies? They worked with Trinity Insight, a consultancy, to help increase their sales and conversions. Trinity Insight used Visual Website Optimizer for A/B testing and then shared their excellent case study with us. (Key point to note is that no internal IT resources of SmileyCookie were used in the test. All of the following was done using Visual Website Optimizer interface).
A/B test challenge
SmileyCookie has a large header location which they use for seasonal or promotional offers. It is at a prominent place and hence catches attention of the visitor. Since an eCommerce store can position itself in various different ways, Trinity Insight wanted to test which value proposition works best for SmileyCookie visitors. They wanted to understand which aspect about SmileyCookie’s service is more important for visitors. Following is a screenshot that shows the header where these value propositions were tested:
Trinity Insight tested five different value propositions and designed their respective creatives. These were five different value propositions:
- Order Today -> Ships Next Business Day
- Want to save $5 OFF your next purchase? SIGN UP NOW ->
- $6.99 Ground Shipping For Your Entire Order
- FREE SHIPPING on any order over $40
- Cookies Made Fresh & Hand Iced For You!
Following screenshot depicts the creatives that were developed:
A/B test results: 41% increase in sales
While the test was running, the overall conversion rate for the site was 8.98% with an average order value was $29.85, yielding a $2.68/per visit value. Over this same period of time, the winning creative, Order Today -> Ships Next Business Day, had a 12.61% conversion rate and a $29.95 average order value resulting in a $3.78/per visit value, a 41% increase over the site average. Note that this increase was statistically significant at 95% confidence level.
Here’s what Trinity Insight said about the results:
Prior to testing, SmileyCookie could only make educated guesses as to which core value resonated with their customers and an opportunity to generate an additional $1.10/per visitor would have never been realized. By continuing to look at ways to make the site more engaging to their customers and testing these opportunities, SmileyCookie is guaranteed to create an experience that’s better today than it was yesterday as well as learn great lessons along the way about what their customers truly find value in while visiting SmileyCookie.com.
Why next-day shipping worked?
Cookies are perishable items and when you are ordering cookies as a gift or for special occasion, you want them sooner than later. Next day shipping increased sales because it tells the visitor that they don’t have to wait for long to get the cookies. There is no uncertainty and visitors know exactly when they are going to get the cookies. Since these cookies are ordered as a gift or for special occasions, customers may not be very price conscious and hence value proposition regarding discounts or low shipping rate didn’t work that well.
Of course, all this is obvious after we have A/B test results. And in fact, all the reasons proposed above may not be true. Interpreting A/B test results is always hard because if you know why a variation works better, you don’t need to do A/B testing in first place. You can simply implement that variation directly without A/B testing.
But, nevertheless, interpreting A/B test results are useful to see what would work in future or for other similar websites. So, if you sell other such perishable items online (say cakes, food, ice-cream, etc.) emphasizing on how fast your shipping is may increase your sales. Try A/B testing it!
Hope you liked this case study. If you have any comments or feedback, let us know!
More optimization awesomeness
Know all that you need to get started:
- What is A/B Testing?
- Is it compatible with SEO?
- How to start your first A/B test?