and so should you...
It has been criticized, but it is always guaranteed to work. What is that we are talking about? Yes, you guessed it right: red buttons! No matter how many people consider such A/B tests as a trivial exercise, every now and then they have been demonstrated to increase conversions.
The theory of red buttons also worked for one of the Visual Website Optimizer’s users, although they use orange, which fits their color scheme, instead. GSM.nl is one of the Netherlands’ largest eCommerce shops selling mobile phones, GSM plans and other mobile accessories. As you can imagine for an eCommerce site, they have Buy Now buttons used all over the website: product pages, catalogue pages, special offers pages, etc. The challenge for this particular A/B test was that they had to vary ALL buttons on the site at once. A lot of pages (such as the homepage) contain multiple instances of the order button, one for each featured product. This seemed complicated, but with Visual Website Optimizer they designed it in a matter of minutes.
All they did was created an alternative CSS stylesheet, and run the A/B test on the different stylesheets. The stylesheet defined how Buy Now buttons looked like, so if they do a split test of stylesheet they will automatically split test ALL the buttons on the website. Clever!
Here are different variations that were tested:
| With text buttons | With green buttons | With red buttons |
 |
 |
 |
The test results showed that the red (well, technically orange) buttons increased overall website engagement by 5% (statistically significant). Engagement is defined as click on any link on the page, so an increase in engagement means a reduction in bounce rate. Sales were also measured as one of the goals, which showed an increase too, but due to the relatively short test period, did not yet prove statistically significant)
As a follow up, of course, this test will be run long enough to determine if button color has any effect on actual sales. We are guessing that it makes a small contribution to increased sales, as more people use the site actively, but taken into account that a decision to purchase a product involves numerous variables such as product cost, shipping costs, discounts, etc, it is hard to measure if the change in color is the determining factor.
The case suggests it is safe to say to that a bright button color works well for catching attention and reducing bounce rate, and might even help actual sales. No matter how strong your gut feelings are, ultimately data tells the truth.
In the case of GSM.nl, it proved Visual Website Optimizer could easily test difficult questions, and provide answers in a matter of days.
You Need A Budget (YNAB) is a user-friendly application for personal finance management. The company teaches people how to become awesome at managing their money by following a 4-Rule methodology. And they sell personal budgeting software that follows the methodology. The software is elegant, simple to use, and does just what you need it do – meaning you don’t bog down in unnecessary features that eventually make you want to quit (and pull your hair out).
Using Visual Website Optimizer, they wanted to increase the number of downloads for 7-day free trial of their personal budgeting application. For their first test, they chose to optimize product tour page. For software applications, the tour page is the one which takes up the job of convincing a casual visitor to try out the application. Motivated visitors will anyway download the application so it is the semi-interested visitors who need to be convinced. To optimize the tour page, YNAB chose to do a simple A/B split URL test with the conversion goal as downloading of the free trial.
They designed the variation with following objectives in mind:
As you will see below, the major change was including many more screenshots and hence not requiring visitor to click on different ‘categories’ to see what app can do. You see, semi-interested visitors are more likely to hit the back button than activity hunt for what your app can do. So, including all relevant functionality of the app in form of screenshots on a single page did the trick of convincing them to download the free trial. Here are the screenshots of variations:
Original (control)
Redesigned (variation) – 85% increase in downloads
The variation with more screenshots and lesser categories/links increased downloads by 85%. This result was statistically significant at 97% confidence, so YNAB has implemented it permanently for their tour page. In their own words, here is what they concluded from the test:
We simplified the page presentation and strengthened the call to action with a very convincing and authoritative quote.
Thankfully, like many other customers, they had great praise for Visual Website Optimizer:
Visual Website Optimizer was extremely valuable. We’ve been working with Google’s website optimizer for the past three years and we’re never going back. We were constantly having to jimmy-rig the script code, make sure it was not conflicting with Adwords code, or Google Analytics, etc. It was a big enough nightmare that we avoided testing as much as we could – something that cost us real money over that time obviously.
Are you wondering what is going to be the next page they are going to optimize? Homepage, of course! YNAB will optimize their complete funnel from homepage to product tour to payment. We wish YNAB best of luck for their future tests.
ReplaceDirect, a Visual Website Optimizer customer, is a successful Dutch e-commerce site. It’s one of the biggest retailers in the Netherlands in the field of parts and accessories for notebooks and other mobile equipment. The main focus is on energy products like batteries and power supplies, but the product range is quite extensive.
They recently did an A/B test using Visual Website Optimizer where the winning variation reduced cart abandonment by 25%. For e-commerce sites, cart abandonment is one of the most frustrating aspects of business. After all, there’s no company that likes a potential customer to abandon the purchase after spending tons of time on site research and selecting the right products. Fortunately, cart abandonment can be fixed using A/B testing and this case study demonstrates how ReplaceDirect did it.
A recent study by Forrester found out that shipping costs rank as the number one reason for shopping cart abandonment. Combined with other reasons found out in the research, it can be concluded that visitors abandon their shopping cart because they are unsure about additional costs or surprised to find out (only after entering the payment process) that they are actually charged more than they expected. In order to prevent such surprises, ReplaceDirect already tells the customer in an early stage that no shipping costs will be added. So the customer is clearly informed right from the start. This is exactly what ReplaceDirect wanted to test: will it help reduce cart abandonment when a clear order overview is provided during the initial steps of checkout?
Another cart abandonment study by Paypal
They tested their second step of the checkout procedure. It’s the page where customers are asked to fill out their personal information and shipping address. ReplaceDirect made several changes to the page. Probably the most important change was the insertion of an order overview, comprising the product(s), total costs and delivery date. ReplaceDirect also mentions several benefits at this page; these were changed to more relevant benefits specifically matching this step in the order process. Finally the layouts of the page and the form were changed to yield a cleaner look. Several fields, which were not absolutely necessary, were removed. Following are the screenshots:
Control page
Variation page: 25% reduction in cart abandonment
Their new version performed significantly better, reducing cart abandonments by 25%. As a direct result of this reduction, they saw a 12% increase in sales (14% by the time of pushing all new traffic to the new page). ReplaceDirect says they expected the new version to perform better, but an increase that high turned out to be a pleasant surprise. Well, it is always nice to have a 14% increase in sales without releasing any new products or without spending money on ways to get more traffic to the website. 
When we asked them if they had any lessons for other e-commerce sites; here is what they had to say:
Try to give the customers exactly the information they need at that particular page or section and leave out the redundant. Always put yourself in the customer’s shoes. This may not be easy so: test it!
ReplaceDirect also had some praise for Visual Website Optimizer:
VWO was very valuable. The ease to set up a test and real-time results made it great to work with.
We are very happy that ReplaceDirect reported these great results by doing an A/B test using our tool.
If you manage an e-commerce site without doing any A/B split testing, you are certainly leaving a lot of money on the table.
These days you have all sorts of trust badges available to put on your websites. Some are SSL badges, some are good business practices badges and some are even carbon neutral certification badges. But do these badges have any effect?
A Visual Website Optimizer user, Karsten Lund (a conversion specialist from Optimeria.com), recently carried out a six-month long A/B test to determine whether an E-mark badge (certifies ethical conduct of Danish businesses) impacts conversion rate. He wrote a detailed case study in Danish and we recommend you to read it here (if you understand the language). For those who prefer to read the case study in English, read this post instead.
What was tested?
Karsten tested the presence of E-mark (the eye-like icon in the image above) on an eCommerce store House of Kids. 50% visitors saw this badge and 50% didn’t. The test started producing great results right from the start but our team consulted Karsten to run the test longer so that he can be really sure about the significance of the results.
What was the result?
Yes, you read it right, a 32% increase in conversions with the E-mark badge. And Karsten ran the test long enough to make sure result was statistically significant.
Interesting nuggets
A lot of other users agree with him, though.
Grain of Salt
Even though this particular result has been fantastic, you shouldn’t adopt trust badges on your website without testing them first. Different types of websites/badges combinations may have different results. Elsewhere on Internet, some report an increase in sales, some report a decrease in sales. So, the best strategy is to test it first and then only implement it on your website.
We are starting a new series to profile testimonials and success stories of users who love Visual Website Optimizer. The idea of I Love VWO series came to us when one of our users, Melvin Ram, sent us a testimonial full of praises (without asking, how fantastic of him!). Following is what Melvin Ram from Volcanic Web Design Company sent us:
I love VWO because it’s frictionless. Within 1 hour of setting up an account, we had 2 tests running. Not planned, but running!
Now that we’re doing testing, we’re making smarter decisions because they are based on real data about the behavior of our prospects. We previously didn’t take the time to track this because we’re a relatively small company and with Google Website Optimizer, setup took way too much time. VWO has already prevented us from making a big mistake that we were planning on trying for a month. Instead, we were able to run a limited test and within a few days had enough data to see that our clever idea wasn’t such as a good idea after all.
Now there is no need for guessing or debating internally. We just run a test and see what really happens. We’re fully committed to testing & conversion optimization and VWO is our tool of choice because they are the only affordable solution that we know we’ll actually use all the time because it’s so damn easy.
Even though we’re a relatively small company, our ambitions are bold. With VWO, we know we’re outsmarting our competition instead of outspending them. That is why I love VWO!
If you too have a story or testimonial to share, please email it to love@wingify.com. We will definitely get it featured here.
We are in the process of finalizing pricing for my startup Visual Website Optimizer, which is an A/B and Multivariate testing tool. As you can imagine, fixing price is one of the toughest decisions that a startup has to (inevitably) take. Once fixed, it could be extremely difficult to change it without annoying a lot of customers. We want to be extra sure that we don’t end up under- or over-pricing Visual Website Optimizer. So, how do we decide what to charge?
Asking beta users can be one of the strategies and we actually used that for VWO. However, it turned out to be not the best of our ideas because users actually correlate price with quality. Further, if a product is innovative (like VWO is), users aren’t able to rationally determine its price as they would do in established markets (read this research paper [PDF]). Hence, asking users what a new product should cost yields half-baked information as they have little or no reference points to determine the ideal price.
Another way to determine price is, of course, to look at what competitors charge. In the world of A/B testing, there is a lot of variation. While Google’s basic testing tool is free, Omniture and Webtrends solutions easily run into thousands of dollars per month. To be sure, VWO is not competing head-to-head with a free tool – it is a sure shot way to death for any startup. We are actually competing with pricey tools, at least in terms of functionality. In fact, our users’ feedback confirms our hypothesis that VWO is much better than any other A/B testing tool in the world. Does that mean, like enterprise tools, we should also charge thousands of dollars per month?
So, here we are: still undecided about the pricing. We don’t want to charge too high, neither we want to charge too less. To get better insights into pricing, I decided to research if there had been a successful startup in a complementary industry which was in a similar position. I was lucky enough to find Clicktale, a company which provides heatmaps and usability testing tools. Thanks to Archive.org, a great way to trace their progress over years is to look at their website and pricing evolution over time. (Note: I don’t have any insider information about their company nor I am affiliated to Clicktale in any manner. All material that I have used for research is publically available on the Internet).
Clicktale started in May, 2006 as a closed alpha. Within a month, they quickly transitioned to beta and did not exit from beta until May, 2007 (a full year in beta). However, they still remained invite only till July, 2007. Within this beta period they started taking feedback from their users on what they can pay for the tool. Read the thread titled ‘Pricing for Clicktale’ on their forum (dated Feb 14, 2007). Every reply on this thread is a must read, however the poll results summarize their users’ feedback on pricing:
All startups in beta take note from the poll above – chances are that more than half your beta users won’t like to pay for your product. In our survey too, many of VWO users either did not want to pay or wanted to pay a very small amount $9 or $15 per month.
In the forum thread above, two contrasting opinions are interesting:
One user thought that higher pricing is justified as the tool is a direct value addition to business. He also rightly pointed out that the target market is online business owners, who are already conditioned to pay for services. To quote him from the thread:
So $99 is really low priced. If someone cannot take the $99 or even the $49 ClickTale product, and use the information to at least create a positive ROI every month, they probably do not need the product in the first place.
Then there is a user who thought $49 was too pricey for him and $19 was too limited. He compared the offering with Google Analytics, which is free. To quote him from the thread:
As to Adam’s comments [the user who argued $99/month was fine] about pricing, I think he and I are not operating in the same world.. I use Analytics for free, and am paying $19/month for crazyegg. I don’t use enterprise-price-level solutions and I think probably a lot of your customers don’t either.. If you are going to be in the enterprise world, just hire a ton of people for sales & support and quadruple your prices..
This contrast is super-interesting to us because, like them, we are in exact same dilemma – which market to cater? Do we cater to small/medium and enterprise sized businesses who can pay for the service? Or do we cater to long tail of freelancers or tiny businesses who want the service for free or at an extremely low cost? As a startup, the decision of identifying target market pretty much determines our survival. A no brainer: price it too high and we don’t get enough customers. Price it too low and we don’t get enough revenue to cover costs.
Clicktale finally launched to public in August, 2007. Their pricing at the launch time is interesting:
The pricing is same as they had proposed in the forum, however interestingly they increased the page recordings per month (the limiting factor for different accounts) by 5x. In the forum post, they proposed 80 recordings/month for free account, while on launch they increased it to 400/month. This tells they listened to feedback from their users carefully and incorporated it into their final pricing. Take note of that, startups.
In December 2007 (about 1.5 years after they launched alpha), they secured venture capital and the pricing of their product remained the same for next 7 months. In August 2008, they bifurcated pricing plans into personal and business editions. In personal plans, they offered a super-cheap plan of $9/month (10 times cheaper than the existing $99 plan):
And for business edition, they introduced a super-expensive plan of $999/month (10 times costlier than existing $99 plan):
Unfortunately, Archive.org does not have any entry of their website after August 2008. I searched on Google to get a hint of what happened to their pricing after they launched personal and business plans. The latest change in pricing that I was able to track down is in Feb 2009. A blog post reviewing them reproduced their pricing chart:
You can see they dropped bifurcation of plans and decreased the number of choices significantly. They also dropped cheaper plans ($9, $49) and some expensive ones too ($490, $990). They have exact same pricing even today (as of April 2010).
The reason I invested time doing all this research is to help startups such as mine take an informed decision on how to price their products. It is not very obvious that a successful company such as Clicktale, within 2 years, transforms the highest priced plan at the time of launch into their lowest priced plan today. They tried all sorts of pricing plans and apparently found that businesses which use their service value it enough to pay at least $99/month.
As a startup about to get out of beta, what I read from this is that if you provide a product that your users value, don’t fall into trap of selling it at rock-bottom prices (in hopes of compensating it from large volume of users). Another lesson that I take from this evolution is that it is OK to change pricing if your current plans don’t turn out to be ideal.
If you have any feedback, comments or thoughts, please share them with me. I hope you enjoyed my investigative journalism
Answer: none. Using Visual Website Optimizer on my blog, I tested the layout of sidebar to see if it has any effect on bounce rate. Particularly, I tested whether the sidebar in right (default on many blogs) works better than the sidebar on left. The answer for my case turned out to be that it doesn’t really matter. (Though it may differ for you – you should A/B test it, it is really quite simple). Here are the versions I tested:
Control – Sidebar on the right (click to expand)
Variation – Sidebar on the left (click to expand)
Both versions had surprisingly similar engagement rate of around 22%. Engagement rate is inverse of bounce rate, so in a way this means , irrespective of sidebar positioning, bounce rate of my blog remains 78% (quite high, by the way). If you want to see actual results, here they are (click to expand):
Since the test involved changing the layout of blog across all posts, if you are curious that this would have involved digging into Wordpress or PHP code – the answer is no. In fact, the took just 5 minutes to implement. Using the advanced code mode of VWO, all I did was to define the Left Sidebar variation with this CSS: #sidebar { float: left; }. Yes, just that! VWO took care of the rest.
RIPT Apparel is a Chicago-based online retailer of one-of-a-kind designer tees and wearable art. They sell one new design by a new artist every day starting at midnight CST for $10 and it is only available for a period of 24 hours. After that period, the design rests in peace forever in the T-shirt graveyard and a new one takes over its place. The sense of urgency created by limited availability of the design can be used effectively to drive sales and that is what RIPT Apparel did in their first A/B test.
They had never done A/B testing before but were intrigued by the concept and wanted to give it a shot. The tool they selected for this purpose was Visual Website Optimizer. They tested their current buy button against a new buy button to determine which would have the best conversion rate. The conversion rate here being sales of their T-shirts.
The following is their control version (or default). Observe that the button isn’t the first thing to catch your attention as it blends well into the overall color scheme.
Original Version
They saw an opportunity for improvement, so they replaced their existing button with an attention-grabbing button with a different color scheme (green). See version A of the button below:
Version A – Control
Much to their surprise, they immediately saw a rise in sales. Ideally, they should have A/B tested the new version against the old but this was their first time and as you will read further they did a great job overall. Encouraged by the results, they went ahead and created another variation of the button. This time they decided to emphasize on the tactic that they should have used all along: emphasizing “limited 24-hour availability”. It has been demonstrated again and again that a sense of urgency drives up the conversion rate and this time too it proved to work best. Following version (B) of the button increased their site sales by 6.3% (notice only the button was changed – no new offers, no new products, no new policies).
Version B – 6.3% increase in sales
Pause for a moment and think what really happened in this case study. Someone (RIPT Apparel) with no previous experience in the field decides to A/B test, spends time researching what could work better, designs button variations, uses Visual Website Optimizer and sees a 6.3% rise in sales. No gimmicks, no new offers, no change in company strategy. Simply testing of sensible better variations of buy now button.
That 6.3% rise in sales will be hopefully permanent, so this one time effort would pay back RIPT Apparel many times over for time to come. Here is what they had to say after the test ended:
“According to Visual Website Optimizer we found our conversion rate improved a whopping +6.3%! We were astonished by this result. We had no idea that changing one button could result in an increase in sales and have that much of an impact. We wish we found this tool much earlier. “
It is important not to forget the role of the tool here. Visual Website Optimizer has been designed with a sole focus on ease of use, while avoiding technical hassles as much as possible. When we asked about the role of the tool in their first A/B test, RIPT Apparel said:
“The sales we have seen since changing and experimenting with our buy button have proven Visual Website Optimizer invaluable. I had some issues setting it up due to our shopping cart system and the team at Visual Website Optimizer walked me through step by step in order to set it up correctly.
As a result of this test I was able to go back to my partners and show them the results, which they were very happy with. Version B. is now live on our site and sales continue to rise.”
A great thing about tasting success with A/B testing is that it pumps you up to do even more A/B testing. So, RIPT Apparel are all set for their next phase of testing:
“We learned to test and test often. We think now that this test is over, we are going to try and test a few different color options, maybe even try some wording choices for our buy button.”
RIPT Apparel is a great company that successfully proved that A/B testing is not something that only Fortune 500 companies can do. Even small-medium sized online business can optimize and increase their sales by not mistaking A/B testing as a hyped-up fad. A/B testing works and it is addictive; one simply needs to try it at least once.
By the way, you should check out RIPT Apparel’s T-shirts – they are really quite cool!
Quick, imagine that you own a small local business, say an automotive repair shop, and you are looking forward to using Internet for getting more business. While researching for your options for tools or services, which of the following pitches is going to appeal you most?
Having difficulty making your mind? You are not alone. CityCliq provides low-cost, search engine-optimized webpages for small and large businesses. As they provide a suite of tools to businesses for creating and managing their online presence, getting the positioning right is critical. They recently used Visual Website Optimizer to A/B test which product positioning works best. The conversion goal for this test was a click on their pricing plan. Implementation wise, this simple test was quick to setup (took them just 15 minutes to do so); major effort was invested in coming up with good candidates for positioning. They ran the test for 2 weeks on their homepage and were really happy with the results.
(Click to Expand)
Here is how different positioning statements fare against each other:
| Pitch | Conversion Rate | % Improvement over original | Statistically Significant? |
| Businesses grow faster online! (original) | 25.3% | - | - |
| Create a webpage for your business | 47.8% | 90% | Yes |
| Get found faster! | 31.8% | 26% | No |
| Online advertising that works! | 20.2% | -20% | No |
About the results, here is what CityCliq had to say:
“Create a webpage for your business was our last test tagline and eventually showed a 98% chance of beating the original. In retrospect, its success makes sense, as it’s perhaps the purest, most direct representation of our product. It didn’t surprise me, as user behavior seems to vary wildly on the web. Plus, there hasn’t been much research done on the behavior of our target audience, so these results were more informative than surprising. “
What they mean is that this A/B test yielded more than just an increase in conversion rate. The test informed them about the psychology of their target customer. This is a perfect example of how A/B split testing can be used to peek into the minds of customers to see misalignment of what they want and what they think you offer. Create a webpage for your business as the winning variation tells that CityCliq’s prospective customers are already aware of benefits of being online and in fact are looking for a tool make that process simpler. All other variations tried to pitch the advantages of being online, which clearly isn’t that appealing to the customers.
Another point to note is that all but the winning variation had an exclamation mark. It would be interesting if they follow up the test with two versions of the winning variation – one with exclamation mark and one without. After A/B testing was completed, CityCliq implemented the winning variation permanently on homepage (as you can see on screenshot) and subsequently saw a rise in user actions. Inspired by their first successful test, they hope to setup a lot of A/B split tests in near future.
AquaSoft was founded in 1999 and is a pioneer in photo presentation software. Their products cover a wide range from slide show creation, desktop publishing, photo books, etc. They recently candidly shared details from their latest A/B testing project, where they used Visual Website Optimizer for increasing sales from their eCommerce site by 20%. Here are the details on the test as told by AquaSoft:
Question: What was the conversion goal of the test?
Answer: The goal was to increase the sales on our product overview page. This page already had a higher conversion rate than the rest of our pages because people who visit this page already are interested in buying.
Question: On which page did you run the test?
Answer: Shop Overview (listing of all articles).
Question: Which part of page did you select for the test and what variations did you test?
Answer:
Original version (click to expand)
Redesigned version (click to expand)
Question: Why did you think that the variations you created had better chances to beat the original? What were you actually testing in this test?
Answer:
Note that we did not add any new feature nor the guarantee was new. It was just not described on the old page.
Question: What results did you get? Were you surprised by the results?
Answer: Improvements started with about 50% but settled on about 35-40% after some time. I was very happy with the result but didn’t trust it. What was measured by VWO were the clicks into the shopping cart, not the actual sales.
So I measured the sales with the funnel visualization of Google Analytics which showed an increase of 17.7% of actual sales. It also proved that the number of clicks into the shopping cart were about the same what VWO had measured. It shows that the bounce rate within the shopping cart was somewhat higher in the winning combination but the overall increased click rate outtakes the bounce rate.
In the second phase, we removed some whitespace and moved the product name above the box shots and gained another 10% increase in clicks and the final increase in sales settled at about 20%.
Question: Any lessons which can be derived from your test?
Answer: Don’t think your website is good as it is. Always test, always improve. It’s a slow process but worth doing.
Our concrete test shows that a clear modern design improve sales. Especially in the sales process trust building is very important.
Question: How valuable was Visual Website Optimizer for this test?
Answer: Is was so incredible easy to create the test (A/B split test in this case) that it takes nearly no time. Because we use an external shopping cart we can not measure in the cart itself so the feature of VWO tracking a click on a link without loading the page made it possible to run the test at all.
Subscribe blog via RSS
