What is A/B Testing? An Introduction to Split and Multivariate Testing.

It’s hard to admit we’re not always right.

It’s sometimes difficult to be honest with ourselves and realise our years or even decades of knowledge and experience still has the potential to bring about decisions that don’t work in our favor, or indeed, the customer’s favor.

As a result, as digital marketers, as web designers and website managers, we often plough ahead, ruthlessly tweaking our websites and truly believing we’re making a difference to the customer’s user experience and therefore improving conversion.

But are we?

How do we know if what we changed yesterday helped us or hindered us? How do we know there isn’t a better way to do what we plan to do today?

With A/B and multivariate testing, we can test our changes first. We can analyze the results and see which variant works best, if any work at all. We can be 100 percent certain that the changes we make to our website are the right ones, and we can scrap all the wrong ones. And the best part is, there are tools out there that make this incredibly simple and straightforward.

It’s far too easy to sit back and pretend our ego knows best without actually analyzing our changes and deciding what works and what doesn’t. But my decades of experience tell me that, despite the knowledge I have and I’m sure you have too, I don’t always know best, and when I conduct an A/B or multivariate test, I’m often shocked which combination of changes produce the highest conversion rate.

What Are A/B and Multivariate Tests

A/B and multivariate tests work in different ways, which we’ll talk about below, but their end goal is one and the same. They allow us to test various changes to our websites and play them against each other, while a tool automatically analyzes the results to see which variance converted at the highest rate.

It all sounds very complicated, but with many tools out there available for us to use, it’s as easy as us deciding what we want to test, deciding what percentage of traffic we want each variance to receive, setting a conversion goal (ie. what are we trying to get our visitors to do more of by testing different variances) and then sitting back and waiting for the tool to gather the results.

A/B Tests


A/B tests work by playing two options against each other. Option A and option B. Option A is your control (ie. how the website currently is) and option B is your new idea, the option you want to test to see if it performs better than option A.

For example:

I have a search box on my website that allows customers to search for a hotel in their city of choice. Normally, I have the search box placed vertically at the right side of my home page, but I’d like to test it down the left of the home page to see if it improves my conversion rate.

Here is how the test would be setup:

  • Option A: Control (Search bar at the right)
  • Option B: Test Case 1 (Search bar at the left)
  • Conversion Goal: The ‘search’ button being clicked and a visitor hitting a list of search results
  • Percentage of Traffic to each option: 50%

This would mean each option in my test receives 50% of my website visitors, an equal split. The tool I choose to use would automatically make this happen.

How fast statistically valid results are determined by the tool will depend on how much traffic your website receives, but don’t worry, if you only get small levels of visitors, most tools can automatically predict and estimate a variation winner for you.

Multivariate Tests


Multivate tests work in a similar way to A/B tests, but instead of just having a control plus one variation, we can have the control, plus multiple variations.

For example:

Throughout the booking form of my website, where a customer fills in a form to book a hotel, I have a small section that tries to upsell customers on an additional product. This section looks something like this:


I make excellent profit on this product, but it isn’t something everybody wants. Approximately 10% of my customers who book a hotel go on to buy this product too. But what if I wanted to improve on the 10%, and turn that 10% into 15% or 20%? I could run a multivariate test that adjusts the layout of that area of the page, changing font colors, font sizes etc.

Or, I could run a multivariate test that adjusts the wording of the title, introduction and buy now button, to see which combination gives me the highest conversion rate.

You may have three different variants like so (the differences between variants are in bold):

Option 1 – Control


Option 2 – Variant One


Option 3 – Variant Two


The list of variants you could have is endless, depending upon how many areas you want to change. Again though, the tools that are out there will work out all the potential variants for you, instead of you having to manually create them. Just decide which areas of text you want to change, decide how many variants of that text you would like to test, and the tool you use will create all the different variants available while sending the correct percentage of traffic to each.

The Benefits of A/B and Multivariate Testing

A/B and multivariate testing tools are often a paid for service, albeit at a reasonable cost. As a result, it’s very important to know the benefits of using such a tool and to know exactly what you’ll be getting for your money.

Trust Me, You’re Often Wrong. Don’t Assume You Know Best.


With our experience behind us, it’s very easy to imagine we always know best. It’s easy to believe we can predict the thought patterns of our customers. But the truth is clear; trends fluctuate. What worked on one website last week, may not work on your website this week.

It’s absolutely vital, then, that we test and do so not just once, but continually. I’ve lost count of the number of times something that seems so obvious to me as being the best solution for my customers is proven with split A/B testing to actually be the worst solution, and vice versa.

Don’t just test things you’re not sure about. Also test things you believe you’re certain about. You’ll be surprised by the results, I’m sure.

A Small Conversion Uplift Can Result in a Large Increase in Profit


The excitement with A/B and multivariate testing often comes in the smallest of packages. If your website receives a lot of traffic, or your products have a high profit margin, even tiny percentage increases of 1% or 2% over the course of a year could well mean large financial rewards.

If I told you your business could see a 5% increase in sales without spending a fortune on marketing, would you be excited? This is the power of testing and tweaking your current website. It’s one of the many fantastic advantages of dynamic media such as websites, rather than media of a static form such as a print advert in a newspaper. To test and tweak a newspaper advert would take several iterations costing many thousands (or a clever marketing campaign that allows you to track multiple campaigns from one print publication). With a website, changes can be made quickly and easily, and tests can be turned on or off with the flick of a switch.

Happy Visitors Become Happy Customers

While ultimately the point of A/B and multivariate testing is to increase visitor to customer conversion, it goes without saying that one of the best ways to do this is to improve the user experience for the visitor.

It was once standard practice to use multivariate tests to test increasingly pushy marketing messages, to change the color of ‘Buy Now’ buttons so they stood out, and other such ‘in your face’ tactics. Increasingly, however, companies are using A/B and multivariate testing to improve website functionality to provide a better user experience to their visitors.

Creating the best possible user experience results in happy visitors, and this in turn converts visitors to customers.

Avoid Waste By Removing What Doesn’t Work

It’s very easy to focus on keeping and improving the areas of your website that work well, but why would we keep areas that don’t work well? If an area of your website is serving no purpose or is causing more harm than good, using A/B testing to find a variation that does work is often an excellent way to remove the clutter from under a customer’s nose. Our website visitors generally have a clear goal. That goal will vary from website to website and from customer to customer, but anything that doesn’t serve a useful purpose will only harm your conversion rate.

The Disadvantages of A/B and Multivariate Testing

While there are many advantages to testing, as discussed above, it’s important to be careful and to ensure we analyse and digest results in a useful way. There are two very clear disadvantages to testing in these forms, but with a clear thought-process and forward planning, these disadvantages can be well mitigated to ensure you achieve the best possible results from your test campaigns.

You’ll Sacrifice Conversion During Testing

When you run tests, especially multivariate tests, you’re not guaranteed that every available variant will result in an uplift in conversion. In fact, while some variants will improve conversion, others may reduce conversion.

A good testing tool will automatically switch off poorly performing variants after X amount of time, should you wish it to do so, but that doesn’t stop the initial hit during testing. It can be well mitigated by looking at the long term goal. If you can find a variant that improves your conversion by 5% for the rest of the life of your website, that benefit is going to far outweigh the slight drop in conversion over the couple of weeks a test was live.

Your Tests Measure Quantity, Not Quality

While numbers suggesting a spike in conversion may seem, on the surface, very positive, we must always remember that these tests purely give us a number, and don’t measure the quality of those conversions.

For example, I have a website that requires people to sign up in order to order my product. I decided to try a split test with a variant of removing the signup process. As I expected, conversion rate (orders) for my product spiked without a signup and registration process, which seemed very positive on the surface. But under the surface, I could see the volume of spam orders, fraud orders, and generally lower quality customers had increased. So while the quantity had increased, the quality of my customer base had definitely dropped.

Multivariate Testing Tools

There are hundreds of pre-built tools and interfaces available to allow us to quickly and easily setup A/B and multivariate test campaigns. We’ll discuss three of the most popular below.



Optimizely is the “world’s leading experience optimization platform’, and is used by large brands such as The New York Times, Microsoft, Spotify and CNN.

Optimzely’s user interface is exceptional, allowing even those with no HTML coding experience to set up, amend and report on tests. The tool allows you to shift elements around your page on the fly from within the Optimzely interface, without having to open up your website’s code. It’s as easy as dragging and dropping, but also has the ability to do more complex scenarios for those of you with more technical knowledge. For example, adding custom Javascript into one or multiple variables is an option.

There is a free option available which will allow you to get a very good feel for the tool before you commit to buy in order to take advantage of all features.

View more information.

Visual Website Optimizer


Visual Website Optimizer has a very similar drag and drop user interface to Optimizely, completely taking out the need to know complex HTML.

There is a free 30 day trial available, with a $49/month fee thereafter for their most basic package.

As well as offering A/B and multivariate testing solutions, the ability to report on click analysis using heatmaps is also readily available.

Just like with Optimizely, you can test across multiple platforms and devices including smartphones and tablets.

View more information.



Unbounce is very unique in that it has carved out a niche to specifically focus on landing pages. It even includes a large number of pre built landing page templates, designed and developed by the Unbounce team, which provide a great starting point for business owners who are yet to set up a page, or are having trouble getting their current page to convert.

Unbounce offer a free version with its core features, but it will display an Unbounce promotional message at the bottom of your page. To remove this, you’ll need to start a 30 day free trial and then pay a minimum of $49/month.

Unfortunately, Unbounce is limited to only A/B testing, rather than multivariate testing.

View more information.

What A/B and Multivariate Tests Should I Run?

The A/B and multivariate tests you run will vary widely depending on your website and its goals. Whether your website is a retail website, selling a physical product or a digital product, promoting email sign ups or simply looking for user engagement in the form of comments and social shares will all influence what type of tests you should be running.

With that said, there are several very common tests that it would be worthwhile trying as you become familiar with your chosen testing tool.

Call To Action Tests

The most obvious tests can be conducted on calls to action. Changing the wording on Call to Action (CTA) buttons to try to influence visitors to take action is often very interesting when you review the results. For example, changing wording of a “Buy Now” button to “Purchase Today” or “Buy Now – Don’t Delay”, could help give visitors an additional prompt and push to convert them to paying customers.

Try changing the styling of call to action’s. Try buttons versus standard hyperlinks. Buttons with icons versus without icons. A green button versus a red button. The list really is endless here and it’s important not to become too granular, especially if your website doesn’t see high traffic volumes (because trying to gain statistically valid results on such a subtle change with low traffic volumes is very difficult).

Content Tests

The copy on your website is often the single biggest converter. The words you write are your chance to sell your product or your service, but the audience you attract can and should determine the tone and style of copy you choose. All too often I see website authors write in the tone that is natural to them rather than the tone that is natural and engaging for their target audience. So try different voices in your copy and see which tone improves conversion.

Other Forms of Media

It isn’t just your copywriting that can be tested and improved. Other forms of media, such as images and video, can be a huge influencer of conversions.

Basic tests that can provide significant conversion uplift include testing different image types. For example, showing a product image rather than a lifestyle people image, or vice versa. Depending upon your audience, you may find they resonate with one type more than the other. For example, a family looking to book a vacation will resonate more with a lifestyle shot of a family on a beach as opposed to a picture of a plane. Whereas a 20 something video gamer looking to purchase the latest game will resonate more with a game play video as opposed to a picture of a man playing the game.


Pricing is a really interesting test metric and has many different options to test and analyze. Test offering free trials versus no free trial. Test showing a price versus not showing a price. Test showing the price at its full cost, versus showing the cost as a broken down fee per day, per week, or per person (if your product allows for it). For example, I might sell an online course that lasts 60 days and costs $359. That sounds like a lot upfront, but split that cost per day and suddenly you’re looking at a more manageable cost of $5.98 per day. Also try price comparisons, showing your packages (assuming you have multiple packages) lined against each other with a clear selection of the features included with each.

Are You Ready for Your First Test?

A/B and multivariate testing is often overlooked by small businesses, but can prove absolutely critical to squeezing the most out of our websites.

Remember to quality control any uplift in conversion to ensure the result is the right result for your business. Be prepared to set time aside to set up tests on a monthly basis and analyze the results. But most importantly, be sure to set time aside to implement the winning results. It would be a waste of time and effort to conduct tests and never implement the winners. And remember to continually test, and that includes things you’ve already tested. As your audience adapts, as your website changes, and as your product line, price points and target market change, your winning tests from 12 months ago could be your losing tests today.

And even more important than everything else… remember you’re not always right. Why guess when you can test?

The following two tabs change content below.

Jamie Harrop

Jamie Harrop is a UK based e-commerce manager, front-end developer and writer with 14 years e-commerce and freelance experience.

Leave a Comment