Since the day we released Visual Website Optimizer in beta, we have been seeking feedback on how to bring the tool closer to actual real-world needs. Till now, every VWO account was associated with a single login and it was a hassle to share that login across different people who are involved in setting up tests. Moreover, agencies and consultants wanted separation between different client accounts.
We are happy to announce that all these features have been launched thanks to inputs from our agency/consultant users, especially Peter from GetYouFound, Janco from Traffic4u and Jason from EmptyMind. Here is a rundown of the new features which are accessible under Account Management link in your account:
Create and manage permission based multiple-logins per account
For the same account you can create multiple different logins and can also restrict them to certain tasks only. Here is the screenshot of the login creation form:
Permissions are hierarchical in nature. Have a look at different options available:
Create and manage multiple (client) accounts
If you are running tests on multiple websites or running tests for multiple different clients, managing them can get quite cumbersome under a single account. Now you can create separate accounts for different clients/websites such that selecting an account only displays tests under that account. Have a look at the screenshot below:
Clicking on ‘Test Account’ will switch to that account and you will only see tests under that account.
Multiple logins for multiple accounts
Here is the killer feature: you can create an account for your client and then create new logins for that account only. What this means is that you can create Browse logins for a client and they will just be able to browse reports under their account (and all your other accounts will be hidden from them). This is a great way to share reports with clients without sharing all of your tests.
Of course, once you are done with a project/client, you can also deactivate their account which disables all logins associated with it.
How to access these cool features?
To repeat, you can access these new features under Account Management link in your account or click the following icon on your dashboard:
Feedback, Feedback, Feedback
We cannot operate in the dark and thus require feedback from you to make sure we are moving in the right direction. Try out the new feature, tell us if you like it or hate it. Tell us what modifications or new features you’d like. As you can guess, we are all ears for your feedback. So write us your viewpoint by leaving a comment here or emailing us at info@wingify.com. Excited to hear what you think.
Note: most of this functionality is currently only available in what is known as ‘Multivariate testing‘ option in Visual Website Optimizer. So if you want to use it, we recommend you to use multivariate testing option (it also allows you to create A/B test).
Warning: if the words CSS, JavaScript or JQuery scare you, it is best not to read this alone. Rather, you should take help from your designer friend to help you understand how Visual Website Optimizer has become the most flexible A/B and MVT testing platform ever. Don’t worry: you still have the world’s easiest WYSIWYG interface for creating A/B and MVT tests. It is just that we have added new functionality to make VWO geek-friendly.
Introducing Advanced Code mode
Visual Website Optimizer has a great, full featured WYSIWYG editor (thanks to TinyMCE) for creating variations. Take a look at the WYSIWYG editor below:
It was designed so that users can be saved from all the hassles of editing HTML and uploading images on their servers (WYSIWYG editor has integrated image uploader). However, some of our web designer users gave feedback that the WYSIWYG editor is limiting for them and they prefer swimming in HTML rather than avoiding it. So, what we have now provided an option of using a non-WYSIWYG editor (based on CodeMirror, thanks Eric for the suggestion). To activate non-WYSIWYG editor, simply click on the checkbox on top right of the variation box and you will have a great code editor. Take a look how non-WYSIWYG editor looks like (for the same variation as above):
Like all CodeMirror awesomeness, it offers syntax highlighting (for HTML, styling and JavaScript), tab intendention, bracket pairing and unlimited undo-redo. Initial testers of this feature say that CodeMirror dramatically improves productivity as far as creation of A/B and Multivariate tests is concerned.
JavaScript API and beauty of JQuery
Advanced mode also gives you an API using which you can dynamically modify your variation before it is displayed on the page. Do you see “Insert JS API” button in the screenshot above? Clicking on it inserts the following JavaScript function in your variation:
As it is obvious from the code above, _vwo_api_section_callback[VWO_SECTION_ID] is a function that is called and the variation and control are passed as DOM elements. This allows you to modify the variation dynamically using your own custom JavaScript code and existing website content (control). Consider following use cases:
1. You want to test if including search keywords in the heading increases conversion rate. Here is how you will define a variation:
Welcome to MySite, you searched for keywords on search engine
2. You have an eCommerce site with multiple products and you want to test the phrase “Buy this for $45″ v/s “Buy now for $45″. However the pricing is variable as it is dependent on the product. So, what you will do is to extract pricing from the control and inject it into the variation.
Control can be for a single product while designing. While the test is running, control will be whatever is present on the (product specific) page:
Control:
Buy this for $45
Variation to extract price and use it:
Buy now for $ something
Lots of other examples can be given: greeting returning visitors, ad campaign specific message, testing latest comments or twitter box, etc. In a nutshell, where ever you need to test dynamic content you will find this API useful.
Limits are endless: the API gives you all freedom of server-side testing (where you generate different pages programmatically) with all the benefits of client-side testing (no page tagging, faster creation of tests, no involvement of IT resources, and best of all not touching site code at all).
Specifying CSS path of the elements
Imagine you have a page where the same element (say a button) is repeated at multiple different locations and you want to do an A/B test while making sure all buttons get the same variation. For example, take a look at the following page:
What if you want to test all “Buy” buttons at once (and that too without page tagging)? Also, remember featured products can change all the time so you can’t simply test the whole section. Using advanced mode, here is what you will tackle the problem. Simply select one of the buttons and change its CSS path to a generic one, like the following:
During the test, all elements matching this CSS expression will be replaced by the same variation. Using the same method (by giving a generic CSS expression) you can also present a consistent experience if elements being tested are on different website pages and at different locations.
Testing of style sheets and JavaScript: decoupling website content
Using the same advanced code mode, you can manually define a section which has no HTML content, therefore isn’t linked to any specific website content. Rather this type of section contains style, script or link tags that are injected in a page’s HEAD section. This way you can test different style sheets or themes (defined inline or linked externally) across your website. Or you can use it to have different visitor experiences triggered by variation-specific JavaScript or JQuery code. Here is how you create such sections:
Note that you can also define a normal HTML section this way (the same section that you can visually select).
Once you create a HEAD section, you can define variations using the same CodeMirror editor:
To summarize, advanced code provides following new functionality:
The flexibility that Advanced Code mode provides for A/B and MVT testing is simply overwhelming. This mode enables all kinds of testing and should be a web designer or developer’s delight. We worked hard on this feature and would love to hear feedback from the designer and developer community. Do you like the new feature? What would you like to add to it?
Graphical visualization of dense information is perhaps the most important component for metrics-heavy apps such as Visual Website Optimizer. Accordingly, intuitive display of A/B test results is a priority for us. In the same spirit, we’re happy to introduced new test result charts. Take a look at the screenshot below and bask in awesomeness:
On the left Y-axis you can see conversion rates for control and different variations. The red series represents average conversion rate. On the right Y-axis you can see total visitors in the test and total conversions. Also, as you can see, hovering on points pops up a box summarizing that day’s performance of the variation. What is not obvious in the screenshot is that by clicking on series legends, you can switch them on and off. Plus, hovering on series legend highlights the respective series and dims all other series. How cool?
These shiny new charts are powered by an open-source library called Flot. This library is absolutely fantastic and very, very powerful. In fact, it is so powerful that we were able to implement new charts in a single day. Special thanks to @NilsFredrik of Maksimer for giving inputs for the new charts and for insisting that we should use Flot for this purpose. We owe new charts to you, Nils-Fredrik 
Do give new charts a spin and let me know your feedback. If you have any ideas on how we can improve visualization of A/B or multivariate test results, I will be very happy to hear them out. Simply leave a comment on this post and let us start a conversation.
So, do you like new charts?
People usually have lots of unfounded excuses for not doing A/B testing. A major excuse right on top of the list is ”OMG, WE DONT WANT ANY LOST SALES”. Its toned down interpretation means if test variations are not good enough, A/B testing can actually hurt a website’s conversion rate. This reason is a major deterrent for eCommerce (or other kinds of) websites where every sale matters. So, if a variation isn’t performing as well as the control (or deafult), that means sales are getting lost during A/B testing. On the face of it, the argument sounds really convincing.
However, the reasoning is quite weak, and that is because:
a) Your benchmark should be overall site conversion rate, not the conversion rate of control
Poorly performing variations is the price you pay for doing A/B testing. That said, if some variations aren’t doing well, other variations may be doing much better. Even if multiple variations perform badly but your overall conversion rate is good (perhaps due to one or two extremely well performing variations), you should be in your comfort zone. Actually, the fear of poorly performing variations resulting in lost sales is completely irrelevant as long as overall conversion rate (accounting all variations) is higher than your existing (control’s) conversion rate. So, before starting to worry take a peek at the right metrics
b) A secret ninja trick: you can always disable poorly performing variations
Most modern A/B testing tools (including Visual Website Optimizer) provide one-click option to disable variations that you don’t want to keep in the test. If you think a particular variation is not performing well, simply disable it. (Remember: don’t disable a variation at the slightest hint of poor performance – always wait for statistical significance).
c) Still not convinced? You can always let the Visual Website Optimizer monitor the test for you
Here comes the magic part. Visual Website Optimizer has recently introduced a feature called “Risk-Free Testing” whereby (if you choose) your test is monitored in the background and VWO automatically disables poorly performing variations without any human intervention. That means your test is guaranteed to perform as well as your website’s existing conversion rate because all non-performing variations get disabled as soon as they are detected .
d) Okay, you even want maximum conversion rate possible? Sir, you can implement the winning variation on your site automatically
Another part of “Risk-Free Testing” is an option to automatically allocate 100% traffic to the winning variation once a winning variation has been found (during monitoring in background). This dramatically reduces the amount of attention you need to devote to the test as it ensures that the winning variation will be implemented on your website automatically. No IT department involved for post-test implementation and your developers can go on a much needed vacation.
e) In nutshell, your only risk is the time invested. But, hey, look at the upside.
As argued above, “Risk-Free Testing” removes all risk involved in doing an A/B test. Given zero risk, what is the worst that can happen if you do an A/B test? Your time invested can get wasted? Yes, your 20 minutes invested in thinking a new headline or 2 days redoing the sales page can indeed get wasted. However, look at the potential upside: 20% increase in sales, 90% increase in conversions and what not.
By the way, a “no-result” isn’t worthless – it tells that your existing page is doing really doing well and you should avoid tinkering with it. Or it says you are testing wrong sections or your variations are poor. It is actually a lot of information and should give you interesting insights for your next A/B test.
To sum this rather long post, here is a golden nugget for you: What cannot come down can only go up. (And, yes, we are talking about conversion rate here)
Yay! The most requested Visual Website Optimizer feature is finally here. Now you can set multiple goals for your A/B or multivariate test. Earlier you could only define one goal per test, which was limiting for a lot of users. But now the game changes completely with multiple goals.
Suppose you are doing an A/B test on your homepage, with the new feature you can now track the performance of variations on any number of goals which can be of different types: click on a link, visit to a page, form submission, visitor engagement or custom conversion event. This means you can track the impact of a variation on your site’s bounce rate, registrations, sales or any other arbitrary goal you can imagine. Essentially, tracking goals across your funnel during an A/B test has become a lot easier.
Moreover,VWO’s integration with Google Analytics, Omniture SiteCatalyst and other analytics packages will let you track performance of variations on your analytics metrics such as time on site, repeat visits, pageviews/session, etc.
VWO will track conversions, crunch statistical significance and plot charts for all the goals you define. However, there would be a primary goal (which you choose) that decides the winning combination. As an additional convenience, you can now also name your goals so that they become easily recognizable in the reports. What’s more, you can even add multiple goals to your existing tests. So, try out the new feature now – it will definitely provide a lot of additional insights. Or, if you aren’t a registered user, play with the new functionality in the demo.
We are working non-stop to build world’s most powerful testing platform, and this multi-goal testing was a must-have feature towards that path. Do try out the new feature and let us know your comments and feedback. Hopefully, you will like it.
Most likely, A/B testing isn’t a must do activity for you. After all, it requires spending some time thinking what to test and then some more time is required for coming up with interesting variations. (Luckily, VWO has removed all the technical and integration hassles so you save time there). All in all, it requires time and effort investment, whose value and returns may not be obvious to you. Fully agree with you on that front.
However, for most people, the bottleneck is creating their first A/B test. The technique sounds hyped up, but it really isn’t and there is proof of even the first, really simple test yielding extremely positive results. CityCliq, a service that helps local businesses have presence online, created their first test on VWO and saw (statistically significant) 90% increase in conversions. Point to note is that it was CityCliq’s first test, they tested a small headline change, saw results and they are now hooked to A/B testing.
Even when the value of A/B test is strikingly clear to me and companies like CityCliq, AquaSoft and MedaliaArt, there are many who somehow still manage to come up with excuses for not doing A/B tests. One by one, I tackle all excuses and try to convince why you should create your first A/B test right now! Following is the list of common excuses and my perfectly justified counter-excuses:
A/B testing takes too much time to setup (integration and technicalities)
False! It takes less than 10 minutes setting up the test. If you don’t buy this, watch this video or try setting up a test yourself (while keeping an eye on the clock).
A/B testing takes a lot of effort in coming up with different variations
While this is true (and entirely justified) for A/B tests where entire design is tested, this is certainly false wherein you are testing headlines, call-to-action and other small but significant elements on your website. With WYSIWYG editor and, making variations is as easy as working on Microsoft Word (now some may debate even this but I think Word is pretty easy to use).
There are no guarantees that I will get results with A/B testing
True, there are no guarantees. But then there are no guarantees for anything worth its salt. Clichés apart, unless you act on something you shouldn’t expect your sales and conversions to increase automatically. A/B testing is no magic, to be sure. You cannot simply signup for VWO and expect sales to rise. However, if you aren’t doing A/B testing (or other doing business/product improvements) you can definitely expect sales to stay flat.
A/B testing will hurt my existing conversion rates
No it won’t. Most sophisticated A/B testing tools, including VWO, have options to switch off or disable variations that aren’t performing well. So, you don’t really have to worry that your existing conversion rates will take a huge hit in case variations are bad. You can setup a test, sit back, relax and go have a hot cup of coffe.
I know the truth (or, in other words, I already know what will work on my website)
No, you don’t. Trust me, you don’t. There are lots examples of A/B testing where the winning variation was an underdog. Except for systematic testing, you have no way for knowing what will work and what won’t. Of course, if you really know the truth, try doing A/B testing of your existing design against your hunch-based improved design and let your boss, clients, colleagues know how you always knew yours is better. (Warning: this will either shatter your status as an expert or take it to the next level!)
It requires a lot of traffic
Yes and no. The significance of your results depend on a lot of factors, and not just only on amount of traffic. You can play around with this A/B testing traffic estimation calculator and actually see what amount of traffic do you really need for your test. In some cases even a traffic of 300-500 visitors may be good enough. (In other cases though, 10k visitors may not be enough). But if you are testing for dramatic changes on the site, you can expect to get statistically significant results with moderate amount of traffic.
I am not ready for it yet
Why? Have you ever thought when would you actually be ready? When you roll out the product? When you start getting a million visitors each month? When your customer base doubles? Frankly, A/B testing should be done at all the stages of evolution of a website – from startup to establishment. In fact, for startups A/B testing is even more important because in initial days there are so many questions to be answered related to business, market, product, etc. A/B testing helps you find the answers you need. (Like it did for CityCliq).
The key lesson here is that if you have been procrastinating A/B testing for long, you should do it right now. If you have any excuses for not doing A/B testing that I haven’t covered, leave a comment on this post. I will most definitely come up with a counter-excuse why you should be doing A/B tests all the time
If you want to track conversions for an A/B test (created using Visual Website Optimizer) that aren’t on the same domain, you will find this post useful.
How to track conversions on a separate subdomain
UPDATE: now subdomains are tracked by default. You don’t need to do any additional settings or add any extra code (such as the one below).
By default the cookies set by VWO is available only on the main domain (and not subdomains). For example, VWO cannot read any cookies for conversion purpose at “secure.example.com” if the test was running at “example.com”. To enable site-wide cookies, you need to specify your main domain. And to do that, simply add the following code (in bold) at the top of your VWO code:
<script type='text/javascript'> var _vis_opt_domain = 'yourdomain.com'; var _vis_opt_account_id = XXXX; var _vis_opt_protocol = (('https:' ....
Only the code in bold is what you need to add. Rest is the default VWO code. Of course, you will replace yourdomain.com with whatever your domain may be.
How to track conversions in a separate domain (or 3rd party apps)
UPDATE: now conversions on a separate domain are tracked by default. You don’t have to do any extra settings. See this blog post for more details.
No matter what settings we do, on different domains cookies cannot be read. So, we will have to use a trick for tracking conversions on pages present on a separate domain. Take following steps to implement this method:
<iframe src='http://yourdomain.com/conversion.html' height='1' width='1'> </iframe>
That’s it. Now you can track conversions for your A/B tests on subdomains and even on separate domains.
NOTE: This feature has been deprecated in favor of new, simpler JavaScript API which won’t require any tagging. Details soon.
In majority of cases A/B split testing is done on static content on a website. That is, the contents of control (default) and variations do not change during the test. However, in some cases part of the werbsite contents (included in the test) may change during A/B test. Consider following A/B testing scenarios where your test includes:
Traditionally you would have to use server side A/B testing or do complex programming from your backend to do A/B tests on such dynamic content. But with this newly released feature of Visual Website Optimizer, testing even most dynamic content becomes a piece of cake.
How to test dynamic content using Visual Website Optimizer
All you need to do is to add a small HTML code to dynamic parts of your website which is included in A/B or multivariate test. Following code (in bold below) needs to be added to the dynamic parts:
<any_html_element class = “vwo_dynamic_1” > any dynamic content </any_html_element>
Tagging the HTML with class=”vwo_dynamic_1” tells Visual Website Optimizer that the particular content is dynamic so when a variation is chosen, it must reflect the dynamic content in the variation correspondingly.
An example will help in making the concept concrete. Suppose you select the following for A/B testing.
Note that the number 32 can change according to number of comments, hence it is dynamic. So, to do A/B test there all you have to do is tag dynamic part (32) with class = “vwo_dynamic_1” in your code/template that generates HTML. Of course, you also need to add class = “vwo_dynamic_1” to specify where dynamic part will be displayed in the variations.
A couple of points to note:
I hope I haven’t intimidated you with all that HTML code above. Though it may seem a bit complex, this small exercise of tagging the dynamic part of code increases the flexibility of A/B testing phenomenally.
Leave a comment here or email me at paras@wingify.com if you need help setting up a dynamic A/B test. Hope you like the new feature
Quick, imagine that you own a small local business, say an automotive repair shop, and you are looking forward to using Internet for getting more business. While researching for your options for tools or services, which of the following pitches is going to appeal you most?
Having difficulty making your mind? You are not alone. CityCliq provides low-cost, search engine-optimized webpages for small and large businesses. As they provide a suite of tools to businesses for creating and managing their online presence, getting the positioning right is critical. They recently used Visual Website Optimizer to A/B test which product positioning works best. The conversion goal for this test was a click on their pricing plan. Implementation wise, this simple test was quick to setup (took them just 15 minutes to do so); major effort was invested in coming up with good candidates for positioning. They ran the test for 2 weeks on their homepage and were really happy with the results.
(Click to Expand)
Here is how different positioning statements fare against each other:
| Pitch | Conversion Rate | % Improvement over original | Statistically Significant? |
| Businesses grow faster online! (original) | 25.3% | - | - |
| Create a webpage for your business | 47.8% | 90% | Yes |
| Get found faster! | 31.8% | 26% | No |
| Online advertising that works! | 20.2% | -20% | No |
About the results, here is what CityCliq had to say:
“Create a webpage for your business was our last test tagline and eventually showed a 98% chance of beating the original. In retrospect, its success makes sense, as it’s perhaps the purest, most direct representation of our product. It didn’t surprise me, as user behavior seems to vary wildly on the web. Plus, there hasn’t been much research done on the behavior of our target audience, so these results were more informative than surprising. “
What they mean is that this A/B test yielded more than just an increase in conversion rate. The test informed them about the psychology of their target customer. This is a perfect example of how A/B split testing can be used to peek into the minds of customers to see misalignment of what they want and what they think you offer. Create a webpage for your business as the winning variation tells that CityCliq’s prospective customers are already aware of benefits of being online and in fact are looking for a tool make that process simpler. All other variations tried to pitch the advantages of being online, which clearly isn’t that appealing to the customers.
Another point to note is that all but the winning variation had an exclamation mark. It would be interesting if they follow up the test with two versions of the winning variation – one with exclamation mark and one without. After A/B testing was completed, CityCliq implemented the winning variation permanently on homepage (as you can see on screenshot) and subsequently saw a rise in user actions. Inspired by their first successful test, they hope to setup a lot of A/B split tests in near future.
