Posted in A/B Split Testing, Case Studies on May 10th, 2012
Video always has interesting effects on conversion rate or sales on a website. In a lot of cases, video helps in explaining the service or the product, and hence increases sales. In other cases, it may actually reduce conversions (probably because it is not a good fit for that website, or that video is poorly made or presented). Previously, we had featured several interesting case studies about videos. One of them was about A/B testing a video on a landing page, and how it increased conversions by 46%. The other one was about testing ‘Watch a Video’ v/s ‘Get Instant Access’.
The only real way to know what a video will do for your website (or landing page) is to actually conduct an A/B test. And that’s what one of our customers Device Magic did on their homepage. They tested existing version of their homepage (which had a video) against a new version which had a jQuery based image slider instead.
A/B test (video v/s image slider)
Device Magic Mobile Forms allows organizations to quickly and easily build robust data collection systems. Using their API, developers can build rich integrations featuring push technologies e.g. for work orders, proof of delivery, maintenance reports, retail surveys etc. They wanted to see if a video or series of rotating jQuery slides would work best for driving people to signup, so they conducted a simple A/B test using Visual Website Optimizer.
Existing homepage (with a video)
Actually, they weren’t sure that the video was of sufficient quality and they suspected it was too technical. So, they thought maybe slides would present their offering more concisely.
Homepage variation (with an image slider)
Can you guess which version worked better?
Results and Lessons
They measured two goals: % conversion from home page to sign up page and lastly number of signups completed. Initially their control (with video) was beating variation (with image slider) and they couldn’t understand why. But then they let the test run a while longer until they had a lot more data – much to their surprise, the result inverted to variation outperforming control, which is what they had expected. And this result was statistically significant.
The image slider variation increased conversions from homepage to signup page by 35%, and the total increase in subsequent signups was 31%. So, it was indeed true that for Device Magic, image slider worked much better as compared to the video.
A key learning from this test would be to get enough data that one is absolutely sure of the result, and that one shouldn’t be too hasty because some detail may be skewing the result or leading one in the wrong direction. Patience in A/B testing is the key learning here. (We have an A/B test duration calculator online to tell you how long you should run a test before giving up on it).
If you have had similar (or contrasting) experience testing video v/s image sliders, do let us know in the comments below.
Paras Chopra
CEO and Founder of Wingify by the day, startups, marketing and analytics enthusiast by the afternoon, and a nihilist philosopher/writer by the evening!
Tags
I ♥ Split Testing Blog
Stay up to date
Follow on Twitter
Email Updates
Subscribe via RSSLatest Posts
Latest Tweets
- Conversion for e-commerce: Two winning tests for a multilingual online florist http://t.co/Si6qLn6scB #
- RT @siteconversion: Call to Action Buttons: The Ultimate Guide on Which Ones Convert and Why http://t.co/upOX52NBv4 #conversion #
- 5 Third-Party Tools for Reducing Shopping Cart Abandonment via @groovecommerce http://t.co/hOMDVssMvV #
- RT @unbounce: A/B Testing Between Free and Paid Signups: Sometimes Free is Better: http://t.co/MtiATLWOeQ by @wingify #
- Interesting results from our new case study - "A/B Testing Between Free and Paid Signups: Sometimes Free is Better" http://t.co/gmd9jyBFwA #
Write for this blog!
We accept high quality articles related to A/B testing and online marketing. Send your proposals to contribute@wingify.com
8 Comments
May 11, 2012
They really need to understand why the video was winning at the start.
Such reversals/crossovers of test results are common and too be expected, but it doesn’t necessarily mean the slider option was the best option.
Depending on your product/service, campaign or whatever, your best, hottest and usually most valuable prospects will convert early in the test.
Once they are taken out, what you are left with is testing the response of the crud, laggards’ late adopters low value customers.
So it’s critical that you figure out who you want to design for: The early hot prospects or the late low value ones. Usually 80% of the revenue will come from the hot prospects.
May 11, 2012
Great post!
Our customers have been quite successful with sliders, however as you state, testing is the key since this should never be assumed.
Considering the usability aspect, for me, I find if a taboo to auto-start a video, and from the test in this post, it looks like it was to be triggered on click. Another great thing about a slider is that the standard is for them to play automatically on page load, thus stepping the viewer through, with no specific interaction required.
May 11, 2012
i think its less about the content (vid vs image). the first screen is cluttered with tabs and bullet points and the vid is off to the left. the second screen is clean, front, and center – its easy to digest
May 13, 2012
You’re right on the money when you said ‘probably because it’s not a good fit for the website’. It’s all about which one expresses the value of the specific business best. Either way, I appreciate the post because I’ve been biased towards videos lately..
May 14, 2012
I think the mixture of both in content of your website works better and its up to you how you want to present your product but I believe if you want to use video than it should be properly created and reader can easily understand your products.
So, I believe using images or video its up to the author how you want to use it but use it in properly.
May 18, 2012
There’s a big difference in the headlines and the way the eye is drawn to the content as ash suggests, I think this may have contributed to the win – it’d be interesting to see a follow up on a multivariate test including different elements of each in the design.
May 24, 2012
I have to agree with Graham, I would present this as a range of multi-variates rather than a simple split. Too much has changed on the page to define it by a single albeit large element.
Also know what it’s like when a client has a “video” that MUST be put on their site even though it is neither irrelevant to their core business nor engaging for visitors. Finally some testing ideas to present to them!
June 1, 2012
My friends from South Africa are terrible at videos. It does not surprise me to see that image sliders was better than that boring 5 minute demo with Dusan’s voice which sounds like a lullaby.
Sorry, the comment form is closed at this time. To start a discussion, please email info@wingify.com.
RSS feed for comments on this post.