Numerous times our clients and customers ask: how many visitors do I need to test in order to get statistically significant results in an A/B test? Unfortunately, there is no straightforward answer to this question as it depends on a number of factors. In the past, we have provided you with an A/B test duration calculator in Excel and also provided with an online tool to calculate the same. However, how it is calculated is sort of a black magic.
In following presentation, I describe just what exactly happens when we calculate number of visitors needed for an A/B test. As the presentation aims to be a high level overview, I won’t be delving into mathematics of it.
Hope you like the presentation! Please let me know if you have any questions or need clarifications.
Editor’s note: this is a guest post by Jeremy Reeves, who is a freelance copywriter and marketing consultant who has produced millions of dollars in excess profits for his clients. To learn more about Jeremy and how he can grow your business, head over to www.JeremyReeves.com or download his FREE special report “The 3×3 Formula For Doubling Your Profits In 60 Days Or Less” at www.3×3Formula.com
The world of Internet marketing and conversions is changing as we know it. Consumers are getting more savvy. Places like Google are putting a stranglehold on how you can market your products. Competitors are popping up in every nook and cranny.
In short, conversions are plummeting and it’s getting increasingly harder to grow your business online.
Because of this, it’s more important than ever to take a stroll through your business and remove any weaknesses in your marketing funnel.
This article will show you how to do just that.
The first marketing mistake you’re making is that you aren’t displaying your core value proposition to customers both immediately and throughout your entire sales process.
Your value proposition is what makes you stand out from your competitors and what gives your customers and prospects a reason they should buy from YOU instead of your competitors.
(It can also be called a USP)
When I look at most websites, I don’t see the #1 reason WHY I should be on that website. Instead, the big mistake people make is saying things like…
…“we’re the best at X”.
We’ve been in business for 10 years.
Our product is more reliable than our competitors.
Uhhh… yea – SO WHAT?
Those things don’t matter to your customers. They’re SURFACE level benefits. They don’t make you stand out. They don’t immediately give value to the person looking at your website.
If you get it right – your customer should be able to walk away from the computer… go on vacation to the Bahamas and remember your value proposition while sitting on the beach sipping rum n coke’s.
That may sound a little far-fetched… but it’s the truth.
Here’s an example.
My value proposition can be stated like this.
“I help businesses maximize their sales funnels and increase profits by 20% – 100% or more without increasing expenses.
On the other hand – a weaker version would be something like “I help small businesses increase profits”.
That’s what MOST people say.
But it’s not specific.
HOW do I help increase their profits? I do it by maximizing their sales funnel… and I also do so without increasing their expenses.
Let me give you one of my favorite USP’s of all time.
Fresh, hot pizza delivered to your door in 30 minutes or less – or it’s free.
This is what Domino’s Pizza used to jumpstart the business into what it is today. Unfortunately they can’t use it anymore because of legal issues (too many people were getting in car accidents) but the value proposition was so strong that they’re still a very successful business today… even though I personally hate their pizza.
Now here’s what I love about value propositions.
You really DON’T have to rack your brain for months trying to create the “perfect” value proposition.
What you do instead is come up with 3-5 of them and TEST them. Usually in PPC ads.
This might seem insignificant but you need to trust me on this… every single website has assumptions of what works and what doesn’t. I’ve worked with dozens and dozens and dozens of different clients over the years and every single one of them had conversion-killing assumptions on their website that they didn’t even know about.
For example… the vast majority of people these days still don’t think that long-form salesletters work.
But here’s the truth.
They DO.
Awhile back I wrote a guest post on this blog called Anatomy Of A Long Salesletter.
Because of that blog post, SOME of the readers kept an open mind and got in touch with me to create long-form letters for their websites. And lo and behold… I gave them big increases in their conversions and profits. One client right now is already looking at a 200%+ increase in leads.
Those who didn’t? They’re still losing thousands of dollars each month because they’re assuming longform letters don’t work.
Another assumption people have is with videos. Some people think they work for EVERY audience. Some people hate them. Some people think only autoplay works, while others can’t stand autoplay.
One of my clients thought that a video wasn’t going to work for his audience and that we’d create both a video and a text salesletter. The plan was to show the video to cold visitors, and the text to his warm prospects that were on his list.
We did this because in previous tests, text won over video. So we assumed text was going to work again for this particular product.
The funny thing is… the VIDEO actually won. And it won by about 30%. So if we assumed that the text would win – this client would have lost about 30% of his revenue.
The point is this.
TEST EVERYTHING. (And use Visual Website Optimizer!)
Here are a few things you can test that most people have assumptions about.
That’s enough to get you started.
Remember… your prospect is the one who is putting money in your pocket. Not you. When you do ANYTHING on your site – you have to mentally be transporting yourself into the mind of your prospect and adding what THEY want you to have.
This means that your language has to be congruent with them… your pricing has to be congruent with them… your bonuses… and your product. The funny thing is, most people develop products that THEY want to have out in the marketplace.
But if you really want to not only boost your conversions but really gain traction in your marketplace, the products you create should come out of a want and NEED… not what you “think” they “might” want.
So the question you’re probably asking yourself is… how do I figure out what my audience wants?
Let me tell you a personal story which describes this personally.
When I first started my career… I was only a copywriter. These days I do a lot more than just copywriting because I realized everything in marketing is linked together, but back then I only wrote copy.
In probably my third month, I had a client who asked me if I knew how to create html. I said no. He was upset because now he had to find someone else who could take the copy I wrote in a word document and transfer it online.
The next client came to me… and the SAME thing happened.
At that point, the lightbulb went off in my head. So what did I do?
First – I learned html and basic CSS so that I could create salesletters for my clients and hand it back to them in “ready to test” shape.
Second – I started LISTENING to my clients more. These days I actually create new services based on what my clients NEED… not what I want to have out there.
For example I have a service where I create automated webinars for my clients and they have to do nothing but read the script I give them. Normally creating automated webinars takes weeks or even MONTHS of their time… but now I can save them probably close to a hundred hours of time because I do it all for them.
Now… that’s how I do it in my own business. But if you’re selling products and not services, here are a few more options you can use.
First is surveys.
This is a GREAT way to do it. There are dozens of survey companies you can choose so just pick which one you think is best for you.
When you’re setting up your survey you have to make sure to give them some kind of gift for filling out the survey. It doesn’t have to be much. Maybe a $10 gift certificate for your website… or a free report… whatever.
Another thing you can do is simply CALL your customers personally.
First of all it’s going to make a huge impression on them because nobody ever just calls clients or customers anymore and simply asks how they’re doing, how you can help, etc.
But more importantly… people LOVE to talk. They’re going to spill the beans – especially if they know you’re genuinely trying to help them.
And a third way to do this is to have a contest on your blog.
First what you do is put a contest on your blog (as well as send them emails)… and ask them what they need. Whoever gives the best response and comes up with a product you come out with… gets it free.
Now I just want to stop for a second and make sure you realize JUST how important and valuable this is. This is what separates “good” companies from companies who blow past their competition year after year and dominate the market place.
When you understand your customer better you can start getting to know their wants and desires on a DEEP level. And when that happens, you can start to fill in your marketing back-end – if you haven’t already. And if you already have then of course you can make it even bigger and more profitable.
But the real value from having a well thought out back-end is that you can then start optimizing sales funnels, instead of just pages.
Let’s first get a clear look at what I’m talking about when I say “sales funnel” instead of a single page.
There are various different sales funnels you can have, but one typical one we would see online is something like this.
Landing Page > Salesletter > Upsell > Downsell > Thank You Page
Now let’s add in some numbers to each of these pages. Again, I want to make it crystal clear that YOUR numbers are going to be very much different than the numbers I’m about to show.
I just want to show you how powerful this can be.
Let’s start with conversions numbers.
For this example, I’m going to leave out your landing page for simplicity… since for many people reading this, you may not have a step before your salespage.
So in terms of your “conversions”… we’ll assume your salesletter is converting at 2%. Your upsell is converting at 20%. And your downsell is converting at 20%.
Now let’s look at traffic.
We’ll say that you’re getting 5,000 visitors per month to your salesletter. That’s a little under 200 people per day, which is definitely possible.
Ok, next, let’s look at prices.
Let’s assume your front-end offer is priced at $97… your upsell is at $197… and your downsell is at $67.
Now that we have all the “base” numbers – let’s add that up to get your “before-testing” number. Then we’ll give just a 30% increase to each of these numbers and see what it turns out to be.
Salesletter – 5,000 visitors X 2% = 100 sales x $97 = $9,700
Upsell – 100 visitors X 20% = 20 sales x $197 = $3,940
Downsell – 20 visitors x 20% = 4 sales x $67 = $268
Ok… so now we have our base numbers. That means right now you’re making $13,908 per month.
Now let’s look at what happens when you increase EACH of those funnels by just 30%… which is absolutely possible and really not that hard at all.
Salesletter – 5,000 visitors X 2.66% = 133 sales x $97 = $12,901
Upsell – 100 visitors X 26.66% = 26.6 sales x $197 = $5,252
Downsell – 20 visitors x 26.66% = 5.3 sales x $67 = $355
That gives you a total of $18,508 in monthly gross profit… which is an increase of $4,600 per month.
That $4,600 increase per month could EASILY buy you a new employee who can get you more traffic… become your affiliate manager to dramatically increase your affiliate base… or any number of things.
And remember, this is for a SINGLE product. Plus it’s not including customer lifetime value. And I’m not including what happens when you increase the TRAFFIC you get to these pages when you start split-testing landing pages, emails, banners, PPC ads and everything that actually funnels traffic to the money pages. That’s when it REALLY gets exciting and the numbers start compounding.
Now… we’ve been talking a lot about numbers. Your number may be bigger or smaller than what I just went through, but either way it’s exciting.
Knowing your numbers is CRUCIAL to any online or offline business. The reason is because if you don’t know your key metrics, you have no idea how healthy your business is. You might think it’s doing well – just to find out something unexpected happens and all of a sudden the floor falls out beneath your business.
Now, I don’t know your business so I’m not really sure what numbers you need to know. However there are a few very common numbers which EVERY business – online or offline – needs to know.
So let me give you them and give you a brief description of each of them. Then I’ll tell you how to USE this information to grow your business.
My question is… do you even KNOW what your numbers are?
Value Per Visitor – This is pretty basic. You want to know how much each VISITOR is worth to your website. The cool part about this is it’s incredibly easy to figure out. Just divide the number of visitors you had over the past month/year/ or week… by the amount of money you made.
For example if your site had 20,000 visitors last month and you made $20,000… that’s an average of $1 per visitor.
Figuring out that metric is important, but even more valuable is when you break it down into segments such as traffic source, page, and possibly even further like gender, age, and other variables if you can figure that out for your business.
Lifetime Customer Value – This is one of THE most important metrics you can ever figure out. By understanding this metric, you can then determine how much money you can spend on advertising… employees… inventory… and everything else that makes a business tick.
But here’s the little caveat to this metric. Actually, there are 2.
First, you want to know how long it takes for the average person to complete his or her buying cycle. For example if you have a continuity program and are tracking the lifetime value of a person entering that program – you have to determine the average length that person stays in the continuity program… and if they buy other products while they’re in there.
So if your membership program is $50 per month and your average customer stays for 4 months… you have a customer lifetime value of $200 over a 4 month period. Then, of course, add in any extra back-end products they bought during that time.
And remember… you’re going for the AVERAGE of all your customers. So don’t calculate this for one customer… do it for about 100 or so. Or even more if you can. The more customers you average into this, the more accurate the number will be.
Now, the second caveat is that you want to determine a COMFORTABLE timeline in which to calculate this number. For example – some businesses have customers who stay with them for many years… or even decades.
However the reason you’re determining this number is to figure out how much you can spend to acquire a new customer, so you might not want to spend $500 to get a new customer if you won’t make that money back for 5 years.
Or maybe you will… it all depends on your business model and how short or long-term you are in your thinking.
Cost Per Conversion (or Cost Per New Customer Acquisition) – This one ties heavily into the 1st metric which was value per visitor. The reason is because if your value per visitor is $1 and your cost per conversion is $1.50 – you’ve just discovered that you’re losing money.
Here’s an example.
Let’s say you’re doing SEO, PPC and affiliate marketing. For examples sake, let’s say each is bringing in $10,000 per month. That means you’re making $30,000 per month GROSS profit.
And let’s say that OVERALL, you’re spending $20,000 per month which means you have a net profit of $10,000 at the end of each month.
Now – what if you broke that down?
When you break it down, you might realize that you’re spending $2,000 on SEO per month and making $5,000 per month gross. That’s $3,000 net profit.
And your affiliate system is bringing you in $15,000 per month gross – and you’re giving away 50% commissions so your net profit is $7,500. now you’re up to $10,500 NET profit.
But whoops – wait – didn’t I say you were only making $10,000 net profit between all 3 channels? Yep… I did.
Because what you didn’t realize is that you’re making $10,000 gross profit per month on PPC, however you’re SPENDING $10,500 – which means you’re actually losing $500 per month on your PPC campaign.
Now that you have that information – you know EXACTLY where to focus your efforts. In this case you’d probably want to optimize your PPC campaign to get it profitable. Or, ditch it and focus on what’s making you the most money… which is your affiliate program.
This, of course, is a very simplified example. You’ll have to run your own numbers to figure out where you stand.
I hope you’ll take this information and apply it in your business. If you’ve read this far, you obviously care about your business. And that’s great! I suggest going back through this article and creating a “to-do” list based on what you think will have the biggest impact on your bottom-line.
Get ‘er done!
This is a guest post by Jeremy Reeves, who is an email copywriter and split-testing freak. To find out how Jeremy can help dramatically increase your email marketing profits, head on over to http://www.JeremyReeves.com/email-copywriter.html
Due to the nature of my business, I’m exposed to more email campaigns than you could ever imagine.
And I have to admit…
… not ONE of them is fully maximizing their email marketing potential.
Every single email campaign I’ve ever looked at has holes in it which are leaking profits on a daily basis.
It’s a sad, but very true, fact.
So in this post, I’d like to show YOU how to maximize your email marketing by giving you 5 “holes” you can fill, starting today.
It makes sense to make this the first reason, because as a reader of Visual Website Optimizer, you should be a split-testing freak like I am!
Now, I’ll admit that for whatever reason – most email marketing software makes it hard to split-test campaigns.
But does that mean it’s impossible? Not at all.
Here’s what you do.
Before you send out a new campaign/broadcast, split your list into 2-5 portions (depending on the size of your list). Then, start testing things that give you insights into what your market wants and responds to.
For example…
With each test you do, try to test something that will give you insights as to what you can use in the future. Over time you’ll seem to develop a 6th sense – the ability to “predict” what your market will respond to, which leads to higher profits!
Many marketers are afraid of making their list “mad” by emailing them too often.
But think about it.
One of the most essential aspects of creating a profitable business is getting your prospects and customers to TRUST you, is it not? To develop an airtight bond and rapport with them, just like you slowly build up rapport with a new person you meet who eventually becomes your trusted friend.
Keeping that in mind, who would you trust more when they recommend something to you…
A) A friend you talk to once a month?
B) A friend you talk to 3…5… 7 times a week?
I’ve seen amazing things happen when a client starts emailing their list more often. In every single case, the more often they emailed their list – the more sales they made (and yes, over the long term).
But there’s one caveat.
Be SMART about it.
Don’t blast your list offers everyday because they’ll drop off your list faster than you can blink an eye. Treat them like you would a friend, by sending relevant information mixed in with soft pitches to buy your product.
And every once in awhile… create a marketing campaign, which we’ll talk about next.
For some reason, newsworthy marketing campaigns are as rare as a hen’s tooth these days. I’m not sure if marketers are getting lazy, or just forgetting tried-and-true ways to promote their business and generate good buzz at the same time.
Here’s what I mean.
Think about the various ways you can tie a promotion for YOUR business into either something that’s on the minds of your prospects… or even something that happened in your life personally. Here are a few ideas to get your mind juices flowing.
I personally ran a birthday promotion for my own services a few months back – and booked myself solid for about 4 months. (Thanks to the VWO subscribers who took me up on that offer!) It took me about 45 minutes to setup the promotion.
Here’s the cool part.
You can come up with special promotions for almost ANYTHING. One marketer I know created a promotion based around his wife putting him in the “doghouse” for having too much inventory in their garage!
If you spend just 30 minutes brainstorming – I guarantee you can come up with at least 5 ideas, one of which will be good enough to use. Use the one you like best, create an email campaign out of it which sends them to a custom landing page made specifically for that event, and you have yourself one heck of a profitable promotion.
Oh, and one more thing.
DON’T make the mistake and think your prospects are too “sophisticated” for these types of promotions. It’s one of the biggest mistakes you can ever make in your business, which is why I’ve included it as…
If you think your audience is too sophisticated for fun promotions like that, here’s a little secret.
YOU. ARE. LOSING. A. BOATLOAD. OF. MONEY.
Clear enough?
The fact is, I don’t care if you’re selling to drunk, zit-faced teenagers or to CEO’s of Fortune 500 companies.
They’re both PEOPLE.
And what do PEOPLE want?
They want to have fun. They want to be entertained. They want to be engaged. They want drama (even though most don’t realize it). In fact, most people are psychologically addicted to drama. Why do you think soap opera’s and reality TV shows are so popular?
Whatever way you slice it, you need to add more personality in your marketing. Can you feel my personality oozing out from my fingers and onto your screen?
I bet you can – and that’s why this blog post is easy to read and understand (at least it should be!). If I got all fancy pants and started talking in technical jibble-jabble and hardcore psychology… you’d have no idea what I was talking about.
The point is this.
People like dealing with people, not companies.
Remember that the next time you click “send”.
One of THE easiest ways to increase your email marketing profits is to segment your list.
So what types of segmentation should you be doing?
I don’t know your business, of course, but here are a few you can start with.
Buyers – Every business in the world should have a separate buyers email list. There is NO excuse for not having a buyers email list. Previous buyers, on average, are roughly 7x more likely to buy in the future as compared to first-time buyers. They also make up your back-end which, when fully developed, can triple or more your profits. If you’re not segmenting your buyers – I’m sorry to say but in my mind – you’re simply insane!
Demographic/Psychographic/Geographic – This can be done in many ways. For example you can separate men from women, by age group, by political association, by religion, by type of dog they own (for dog training sites of course), or even by country. The possibilities are endless, depending on your business.
How Active – Here’s a quick promotion you can do to both clean your email list and make oodles of money in the process. It’s called a “reactivation campaign” and it’s something direct marketers have used in the offline world for ages.
Here’s what you do…
1) Do a search to find prospects who haven’t opened an email in 3 months, 6 months, etc.
2) Send them something for free, while also asking them why they haven’t been active lately and seeing if you can help them in any way.
3) After the promotion is over, do a search for those who didn’t open/click the email and flush them from your list. Yes, actually delete them. They’re the non-responders and are clogging up your list. And for the people that did respond and/or download the gift you sent them – they’re now “reactivated” and should start opening more emails!
That last promotion I said is a little more robust than that if you want to do it right (it would take a blog post to explain it in detail), but something is better than nothing right?
Either way… I hope you’re the kind of person who TAKES ACTION and won’t simply read this post and do nothing. I just gave you enough information to add literally tens/hundreds of thousands of dollars into your business, if you take action and implement.
So start fixing the holes in your email campaign… or hire someone like me to do it for you if you don’t know how or don’t want to put in the effort.
As Nike would say… “Just DO it”.
No more excuses.
You can easily add 20% – 100% more profits to your bottom-line by maximizing your email marketing… so what are you waiting for?
Previously on this blog, we talked about impact of A/B testing on SEO. Now, in this post we talk about how to calculate ROI for your SEO efforts. This is a guest post by Prashant Puri, who is co-founder of AdLift.com – a niche SEO firm in the SF bay area. Previously he was Head of Global Search for Shopping.com (an eBay Inc Company). You can follow/reach out to him on Twitter – @puriprashant
“Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”
- Anonymous
The sentences above basically sum up the importance of measuring your SEO initiatives – only if you can measure SEO ROI can you improve it. Here’s how you get started with it:
You must exclude your own branded keywords from analysis before calculating ROI. This needs to be done because many users will search your brand on search engines instead of typing in your URL directly. But those searches aren’t due to your SEO efforts, hence they need to be excluded. Use your analytics tool to segment keywords into 2 buckets:
Here’s how you can do this (using Google analytics as an example) –
Once you exclude your brand terms – plot the traffic graph for your non-brand terms.
It’s easy to export this data and calculate the % increase in traffic or revenue (if you have ecommerce tracking in place – highly recommended). Once you have the revenue number you can calculate the ROI from your SEO efforts –
It’s relatively easy for ecommerce websites to calculate this once you have the tracking in place; however, the tricky part is for non-ecommerce websites to estimate ROI on SEO. This is where tracking conversions and setting goals becomes crucial – a conversion might be a sign-up, email opt-in, of just pageviews which drive up your advertising revenue.
Once you’ve identified your conversions and set this up the next step is to attribute a revenue number with each conversion. Let’s say you make $5 RPC (revenue per conversion) – so if you drive 50,000 incremental visits at an average conversion rate of 5% and a RPC of $5 – your revenue generated is 50,000 x 5% x $5 = $12,500. Now that you have the revenue number you can calculate your SEO ROI.
SEO helps in increasing the incremental visits as you continuously invest in content and link development strategies. Another lever that you can pull to maximize revenue is the conversion rate on your website. Visual Website Optimizer is a great tool that can help you A/B test landing pages to improve overall conversion rate.
Other SEO key performing metrics that help you analyze the performance of your SEO efforts are –
1) Number of Keywords driving traffic: This is a great metric to track your overall on-page optimization efforts. This tells you the number of keywords driving traffic to your site. In Google analytics go to Traffic Sources -> Search Engines -> “non-paid” -> under source chooses “keywords”.
2) Number of Landing Pages: Another great metric to track, this tells you the number of unique landing pages that are driving traffic to your site. As your SEO traffic start to grow you should start to see this number increase.
Search Engine Optimization has evolved in to being a highly effective marketing channel – and its exciting to be able to track this very closely!
This is a guest post written by Felipe Wesbonk who is a consultant with Traffic Builders, one of the Visual Website Optimizer certified agencies. They extensively use integration of Visual Website Optimizer with Google Analytics and in this article, Felipe talks about how this integration provides them with many additional insights.
Google Analytics provides many more insights into the performance of your website and online marketing campaigns when using plugins of various useful tools, like Visual Website Optimizer (VWO) and BTBuckets. Also by using the Google API it’s possible to make custom dashboards, in which external data can be integrated. During the GAUC (Google Analytics User Convention) in Amsterdam last month, I gave a presentation about these plugins. In this article I explain the integration from VWO in Google Analytics and the way you can use this plugin in making custom reports.
It is very easy to get the most out of Visual Website Optimizer within Google Analytics. Here is the plugin:
if(typeof(_vis_opt_GA_track) == “function”) { _vis_opt_GA_track(); }
Just put this in the GA snippet and make sure you put this above the _gaq.push(['_trackPageview']);. That’s all you have to do. Now the fun part begins.
With the special Google Analytics plugin of the Visual Website Optimizer you are able to integrate the data from VWO with Google Analytics and the e-commerce module. The plugin puts the data of the various variations on custom variables back into Google Analytics. As default, you will see the VWO data in slot 4 of the custom variable.
When the data is here, all kinds of customized reports can be created. This way you can add some metrics, which you do not see in the standard reports in Visual Website Optimizer. You are able to make custom reports and the analysis in order to decide if the variations give the conversions which are satisfying in terms of reach and engagement. This will get you one step closer to an integrated optimization strategy.
As you can see in the screenshot above, it is quite easy to make a custom report with the metrics you find useful. In the filter you can use a regular expression to make sure you get the right data from the custom variable slot 4.
Using custom reports you are able to see which channels drive traffic to the different variations. Choose the most valuable goals as the metrics. In the Dimension Drilldowns you select ‘custom variable (Value 4)’ as first dimension and ‘medium’ as second dimension. Now let’s take a look at the winning variation, which you put into the filter of the report. You can use the custom report to see which channel is still behind in conversions.
The example below shows that the medium ‘referral’ and ‘affiliate’ don’t convert well. Besides, they also have a very high bouncerate (46% and 80%). In this case I would recommend to take a close look to see which sites (sources) from ‘referral’ and ‘affiliate’ give the most traffic. On these sites, you should change the banners and links to make sure it will fit better with the best performing variation in order to achieve a higher CTR and more conversions from those sites.
This is one of the custom reports you could create to get more valuable insights for the best performing variation, in order to make a more informed decision. If you have any suggestions for custom reports which you can make with the data of the VWO tests, please let us know in the comments!
This is a guest post from Jeremy Reeves, who is a freelance copywriter obsessed with split testing everything from emails, to landing pages, salesletters and upsell processes. You can find more information about Jeremy’s split-testing services.
Let’s face it.
Testing kind of sucks. For most people anyway.
It’s hard to learn… takes a lot of time… and typically requires you to hire a split-testing service provider like me to get the job done right (unless you’re a testing-obsessed fanatic like I am and love doing it!).
But – you also know the financial rewards of it or you wouldn’t be reading this blog
And you know that testing can be the single most profitable investment you can make in your business.
But in order to make that happen… you or the person doing split-testing for your business must be aware of the BIGGEST mistakes I see people making when split-testing.
So in this post I’m going to reveal top 7 split-testing blunders your MUST avoid if you want maximum results in your business.
This is a long and information-packed post… so grab a cup of coffee… sit back… and let’s get started.
What are “whispers” and what are “screams”?
Whispers – These are tests such as changing the font, changing the background color, changing the color of a button, testing something in the footer or very low on the page… and basically anything that has no (or very little) chance of getting you a BIG boost in conversions.
(And yes, I realize that “sometimes” some of these things can give decent boosts – but it’s very rare)
Screams – These are tests that are likely to either completely BOMB… or hit a grand-slam home run. They’re the ones that make you anxious to test, but can produce mid double-digit…sometimes triple-digit improvements. Screams are things like getting your copy completely rewritten, testing out a radically different design, testing radically different price points (e.g. $47 vs. $97 instead of $49 vs. $49.95), or coming up with an unheard of and radical guarantee.
Do screams take longer? Yes. Do they require more energy and resources from you? Yes. But is it worth putting money into hiring someone to test radically new copy or a design? Absolutely!
This is one that 99% of marketers (especially online) are guilty of. They “assume” something will work, so they don’t test it.
Here’s a hint.
Don’t assume anything!
I personally conduct split-tests every single day for my clients, and I’m STILL amazed at just how “weird” some of the tests come out to be.
Think your headline sounds “perfect” and nothing can beat it? I guarantee you – some other headline out there floating around in the ethers of your mind (or your copywriters mind) CAN beat it.
Think your pricing matches your market perfectly? I bet not. You don’t know until you test.
In love with your product name? I bet if you split-tested it using Adwords (which is a neat little trick, and easy to do) – you could find one that converts better. After all, when it comes to testing it’s about money, not ego. Go with whatever gets you the highest conversions.
I’m assuming you already have Visual Website Optimizer (if you don’t, you’re nuts)… so I won’t pound this one in too much as you’re probably more familiar with testing than most marketers.
Here’s the mistake most people make.
They think “testing” goes like this.
Step 1 – You watch the conversions of “X” page for 1 week… or for 100 conversions (or whatever other metric).
Step 2 – You then switch that page and replace it with the new test you’re doing… and watch those conversions for 1 week, 100 conversions, etc.
Step 3 – You compare the data, see which had a higher conversion, and you now have a winner!
{Insert obnoxiously loud “eerrrrr” sound}
That’s NOT how you should be testing.
For a test to be accurate, you MUST have all variations running at the exact same time frame, to the same audience.
It’s that simple.
Here’s another one I see TONS of people make. Mostly, it’s an “ego thing” because they want to brag about the huge increase they got.
Here’s what I mean.
I’ve ran plenty of tests where one variation was the control, hitting 95% confidence, after just a handful of conversions (let’s just say 10, for example). Let’s also assume the losing variation only had 4 conversions.
Does that mean the winner actually beat the control by 250%?
Absolutely not!
I’ve seen tests like this where the losing version came back to beat the one that was winning in the beginning.
Why?
Because you need to run your tests long enough to get accurate results! You need to get past what I call “random coin syndrome”.
What I mean is this.
A regular coin should flip 50/50 over time. But I want you to test this yourself. Go get a coin and flip it 20 times. In most cases, you’ll get very uneven results. Something like 15 heads and only 5 tails. Or 12 tails and 8 heads.
And the SAME thing happens with testing. In the beginning, with under 50 conversions or so (this is a rough number and depends on the conversion number), a lot of the results happen by chance.
My advice is this.
If you’re looking to be accurate by making sure your tests are valid – try to get at least 50 conversions or so before making any decisions – EVEN if you have a winner after 20 conversions.
Oh yes, there’s a flip side to what I was just talking about!
Running your test for too LONG is a killer of momentum (and sales)
Here’s an example.
Let’s say you’re testing a “whisper” as I mentioned earlier. In this example we’ll say you’re testing your background color.
5 weeks after you launch the test… the variations have only a 2% difference in conversion.
What do you do?
Here’s my advice.
Depending on how much income that product is bringing in, 2% may or may not make a big difference to your bottom line. If it’s NOT, I highly suggest going with whatever is winning at the moment (even if it’s not at 95%+ confidence) and setting up a new test that is likely to give you a bigger bump.
Bottom line: Don’t get addicted to testing results. If a test goes on for longer than normal without a nice increase in conversions, scrap that test and move on.
(The same is also true for lousy products, by the way. Get your emotions out of the equation!)
Learning your market from your tests is CRITICAL to long-term testing success. It will help you dive into the mind of your prospect and give you a “knack” for knowing what they want.
Let me explain.
When I write copy or do split-testing for my clients… the first step I do is RESEARCH. In this phase I basically submerge myself into the middle of the market and try to understand their wants, needs and desires.
I then use what I discovered in my research to write about in the copy, or to create split-tests with.
And while that’s great and works like gangbusters… it works even BETTER when you can do your research based off what people are actually doing with their wallets.
Here are just a few examples of what you can learn about your customers.
Video: If you have several products, consider testing video against text. Test several variations of video/text together (AKA a “hybrid”). If you notice that video wins… you can then roll that test out and test it on all your other products. In “most” cases, you’ll have a winner each time.
Guarantees: Test several guarantees for one of your products. Let’s say you have 5 products and all of them have 30 guarantees. Change the guarantee to 90 days for one of them, and see if you get an increase in conversions (you will). Let’s just say you get a 10% bump. Well… if you then roll that out into the other 5 products, you can count on a 10% bump on each of those as well… giving you a total of 50% in total increased conversions spread out throughout your site.
Copy Styles : You have several options when it comes to copy. Hypey versus soft. Long form versus short form. Professional versus personal. And while you may “think” you know what type of copy works the best, it’s impossible to know until you test it. But here’s a tip to get started. In MOST cases, a mix of hype versus soft… long form… and personal… works the best. However this all depends on your business, product, market and offer… so test it. Once you figure out the perfect formula – use that for the rest of your site!
Note: If you don’t think long form copy works anymore, read this post I wrote about the anatomy of a salesletter after you’re done with this article.
Want to know if your ego is getting in the way of your testing results?
Take this litmus test.
Has anyone ever suggested running a test and you said “no, that won’t work”… or “no, my customers won’t like that”… or “no, that will make us look bad”…?
If so, your ego is getting in the way.
Sometimes, that’s fine.
If you aren’t trying to maximize profits and would rather secure a specific mental image in the mind of your prospect (such as a certain design style, etc.), that’s your decision.
A perfect example of this is using “doodles” in your copy. These range anywhere from handwritten notes in the margins… to using a crayon on a piece of direct mail.
MOST people would think they wouldn’t work. That “their customer is too ’sophisticated’ for that”. Yet a company I know tested several variations of a letter they were sending out… and then tested one that was in the voice of the owners young son… written in crayon.
Keep in mind, this letter was going to a clientele that were “sophisticated”.
So how did it do?
Well, let’s just say… it was the single most profitable letter they EVER sent out in the lifetime of their business
Sometimes it’s hard letting go over your ego, I get that. But it’s necessary.
Hopefully this article has helped you ramp up your testing efforts. Or, if you’re hiring someone and having them do it… make sure they understand all of these blunders as well. It can literally make the difference between grand-slams and bankruptcy.
Usually you have a pretty good idea of what to A/B test on your landing page (We have 25+ A/B testing case studies in case you need to read those). But once you exhaust all those testing opportunities, you are back to square one. The most logical step is do a proper planning of next phase of A/B testing. Meanwhile if you want to do quick testing, what do you test? In this post, I intend to compile list of 23 quick A/B testing ideas. Here you go:
If you have any more ideas, please leave a comment below. I have tried to include as many ideas as I could think of, but of course, there is no limit to what all you can test!
Red Gate software runs an annual challenge where they buy a small software company for a million dollars. They list a number of requirements that the software company must fulfill. One of the requirements that stood out was about conversion rate. They said:
If you’re selling your product then it must have at least a 10% conversion rate.
This requirement actually made me say “Wow, that’s insane”. Let me elaborate why.
Conversion rate is percentage of visitors who actually bought something on your website. Let’s imagine there are two websites: one sells product X with 5% conversion rate and the other one sells product Y with 10% conversion rate. Now, here is a million dollar question:
Is 10% conversion rate of product Y better than 5% conversion rate of product X?
It’s foolish to even begin answering above question without considering following factors:
In nutshell, conversion rate by itself doesn’t tell much (unless you have extra information like traffic, sales price, lifetime value, traffic mix, etc.) So a website with 1% conversion rate may not necessarily be worse as compared to a website with 10% conversion rate. Conversion rate in isolation is a useless metric.
Conversion rates are not entirely useless. In fact, they are very useful when seen on a temporal scale. In other words:
If your conversion rate is 5% today, aim should be to increase it to 7% (using A/B testing, etc,) or at least not let it fall to 3%.
So, comparing conversion rate over time makes a lot of sense (but for the same website). Unless you have a lot of other information about your competitors, you should NOT obsess over comparing your conversion rate to their conversion rate and whether it is lower/higher. Instead, you should obsess on how you can increase your conversion rate (since that’s one of the easiest things to make your bank balance fatter).
Note: if you go through our library of A/B testing case studies, you will note that we always talk about increase in conversion rate and not conversion rate per se.
In a previous post, I provided a downloadable A/B testing significance calculator (in excel). In this post, I will provide a calculator which lets you estimate how many days should you run a test in order to obtain statistically significant results. But, first, a disclaimer.
When someone asks how long should s/he run an A/B test, the ideal answer would be until eternity or till the time you get results (whichever is sooner). In an A/B test, you can never say with full confidence that you will get statistically significant results after running the test X number of days. Instead, what you can say is that there is 80% (or 95%, whatever you choose) probability of getting statistically significant result (if it indeed exists) after X number of days. But, of course, it may be the case that there is in fact no difference in performance of control and variation so no matter how long you wait, you will never get a statistically significant result.
Download and use the calculator below to find out how many visitors you need to include in the test. There are 4 pieces of information that you need to enter:
Once you enter these 4 parameters, the calculator below will find out how many visitors you need to test (for 80% and 95% probability of finding the result). You can stop the test after you test those many visitors, but you should never stop earlier than that. You may end up concluding wrong results.
Click below to download the calculator:
Download A/B testing duration calculator.
Please feel free to share the file with your friends and colleagues or post it on your blog / twitter.
PS: By the way, if you want to do quick calculations, we have a version of this calculator hosted on Google Docs (please make a copy of the Google Doc sheet into your own account before you make any changes to it).
Ah! The million dollar calculator. Explaining how it works is beyond the scope of this post as it is too technical (maybe a separate post). But, if you have got stomach for it, below is gist of how we calculate number of visitors needed to get significant results.
The graph above is taken from an excellent book called Statistical Rules of Thumb. Luckily, the chapter on estimating sample size is available to download freely [PDF]. Another excellent source to get more information on sample size estimation for A/B testing is Microsoft’s paper: Controlled Experiments on the Web: Survey and Practical Guide [PDF].
Hope you like the calculator and related resources. Excited to know your feedback and comments!
We already have powerful behavioral targeting and segmentation options in Visual Website Optimizer. Today, we’re proud to announce a brand new feature that will take your website to the next level. We just released a feature that is Holy Grail of targeting and personalization: knowing what a visitor is thinking while browsing your website (and then displaying relevant content to him/her to boost conversion rate and sales).
Yes, yes, yes, we are as excited about this new feature as you are. Once you install Visual Website Optimizer on your website, the mind reading feature uses subtle differences in how a visitor is moving his/her mouse to gauge what s/he is thinking right now. Essentially, you get a report on following four attributes (note that this is an initial list, it will expand as our research progresses):
We’re an A/B testing company and to prove this new feature works, we conducted an A/B test. Our hypothesis was that by changing a webpage we can affect visitor mood and if Visual Website Optimizer’s new feature detects that change, it definitely works. We did this test on our homepage. Here’s original homepage:
- Original Homepage – 
In variation, we tried replacing text with an image that was definitely going to affect visitor’s mood in some way. Here’s the variation:
- Variation of Homepage – 
We did this test for a little over a week and here are the results:
As expected, happiness and fun factor increased tremendously when visitors saw that poor boy’s image on our homepage. Moreover, people started subconsciously associating rain drops in the picture with likelihood that it is going to rain today.
Most surprisingly, there was a sharp drop in trust and total signup rate because of that poor boy on our homepage (Glad we have stopped the test. It affected us financially!) Why did that drop happen? We cannot tell! There are always somethings that remain a mystery after an A/B test is completed.
In a word: mathematics! We use various inputs to determine what a visitor is thinking. The most significant input is visitor mouse movements. The way people move their mouse tells a lot about what they are thinking. Strange but true!
Details of implementation only for nerds and geeks: we used artificial neural networks to train a predictive model whose input is vector comprising of mouse movements in past 30 seconds and whose outputs are various factors like happiness, raininess, etc. The data for training this model came from 6 month long research where we recruited a team of 10 normal people from 4Chan.org and asked them to visit various websites and tell us what they were thinking while they were browsing. Without their knowledge we recorded mouse movements and then simply used our expertise in machine learning to map mouse movements to behavior! Genius, isn’t it?
According to our knowledge, no other company in the world provides such amazing technology. So, why don’t you signup for a free 30 day trial of Visual Website Optimizer and try the new feature without any obligations? We can assure you that you are going to have a lot of fun spooking your visitors
