Multivariate vs AB testing: what’s the difference?

Put an end to the guesswork and boost your conversions with A/B and multivariate testing. Here’s everything you need to know

Marketing can sometimes look like pure magic. It looks even more magical when it’s used to convince swathes of people to convert, drive up sales, increase clicks and boost sign ups. But it’s actually more of a science than you might first anticipate.

For marketing to be a success, there are plenty of variables at play. One of those variables is testing.

Testing is a crucial element of both design and marketing. It is used to inform business and design decisions and provide feedback and information that we can use to make any number of changes or optimizations to achieve the overall business goals.

Without testing something, you’re making guesses and if you want to solve problems, you’ll need sufficient information to do that. Testing gives you that information and ends the hopeless guessing game.

In the world of user experience design, two terms are banded around when it comes to testing: A/B testing and multivariate testing.

Throughout this post, Justinmind will explore both of these testing strategies so that your decisions are more empowered. Let’s get into it.

What is an A/B test?

An A/B test is a test which is carried out to compare the success or failure of more than one variable.

A variable can be anything from the length of copy on a landing page, the color of a call to action button or simply the position of UI elements on a web page.

Typically, there will be two different versions of the same thing that you wish to test. A landing page with two different hero images, two subject lines in an email or two different colors for your call to action button to name a few examples. An A/B test wouldn’t work efficiently if it was testing multiple variables. We’ll get into how you can do that shortly.

Let’s say that you’ve got a homepage and it isn’t converting as well as you would like.

You’ve discovered, through using heat maps, that people hover around your call to action button but aren’t clicking it. Isn’t that the worst?

To remedy this situation, you’ve decided to change the copy to make it more actionable. Instead of choosing copy that you think might work, you can use an A/B test to know what works.

The idea is that you present two different designs to two different sets of people. In the case of a call to action, if one design performs better than the other, you’ve got your answer. How do you know which would perform well in this case? By looking at which had a higher click rate.

Common uses of A/B testing

A/B testing is a relatively cheap research methodology and because of that it’s a very popular way to find out key information which can be used to evaluate your website or mobile app design.

A/B tests can help you to validate any assumptions you have. If your call to action button is green but you’ve read literature which explains that red is a more successful converting color, you can test that assumption on your own audience.

What is curious about A/B tests is that you might end up with information that seems counter-intuitive. Take Vendio, whose sign ups increased by 60% after removing the sign up form. It pays to put your assumptions to the test.

You’ll usually find that an A/B test comes in useful when you want to:

• Test versions of a website
• Headlines
• Call to action buttons
• Graphic imagery used throughout your product
• Sales copy and descriptions
• Page structure
• Forms
• Landing pages
• Navigation
• Pricing

Why you should be A/B testing your designs

Now we know why A/B testing is good practice in business and design. But what are the real benefits of carrying out your own A/B tests? There are a few prominent reasons:

A/B tests can make the decision for you. If you’re a designer and you’re working with a content team (or any other team, really) but you’ve got conflicting opinions about certain elements of a design then using an A/B test can put a quick end to any disagreements. Maybe you’ve read that a pop-up asking for an email helps to convert new users to subscribers to your mailing list faster than just a input box found on your website. Your A/B test can resolve these sort of trade-offs backed up with statistical evidence.

A/B tests costs very little money. A/B tests are cheap. When it comes to testing UI elements on a website or different designs, all you need is the alternative to test. Throw them onto your server and use software like Optimizely or Crazy Egg and then you’re pretty much done. Easy to implement, cost effective and actionable insights — what’s not to love?

A/B tests give you insight into user behavior. An A/B test is measuring user behavior at its core. If you run a test and you find that design A performs much better than design B then you can confidently deploy design A, because your decision is backed up with proof.

A/B testing: when it works and when it doesn’t

The crux of A/B testing is that it tests things which are measurable and have one clear goal. Jakob Nielsen writes that measurable actions include:

• Sales for an ecommerce site
• Users subscribing to an email newsletter
• Users opening an online banking account
• Users downloading a whitepaper

A significant drawback of an A/B test is if you don’t have a large audience to make the results statistically meaningful.

Sure, a company like Amazon can perform hundreds of A/B tests on their homepage and get real results. Millions of people interact with Amazon on a daily basis. But what about the mom and pop bakery or that fancy new startup in town? How does A/B testing help them? Well, it does but it will be difficult to achieve robust sample sizes. There are methods around that, thankfully.

You need time to do an A/B test. A calculator like this one can help you determine the length of time you need to get any real significance from your A/B tests.

A small company with 250 unique visitors a day that wants to boost conversion rates by 10% will need at least 162 days to see any significant result. Startups are usually short on time and waiting nearly 200 days is often impossible.

Another problem of A/B tests is that if you haven’t defined your target group properly or you’re in the early stages of your product, you may not know a lot about your customers. If you’re not sure who they are (try creating some user personas to get started!) then you might end up with misleading results.

Interactive prototyping and testing

Testing doesn’t have to wait until production. In fact, you can validate different design ideas in real time using Justinmind.

When you carry out tests during the prototyping stage of the design process, you’re more likely to save money in the long run and save on unnecessary reworks.

What is a multivariate test?

Now we lead onto the another testing methodology: the multivariate test. What is it? It’s like an A/B test but a little more complex.

Whereas A/B testing is used to test the hypothesis of one variable at a time, multivariate testing is used to test a hypothesis of multiple variables which are modified. This is done to uncover which combination of variables performs the best out of all possible combinations.

To simplify: multivariate testing will change multiple variables (for example, longer copy and one hero image vs. two images and short copy) and find out which one comes out winning.

Multivariate testing can be used in the same manner as A/B tests but can be more comprehensive because it will discover the ideal combination for achieving your goals. Instead of testing one variable, you’re testing multiple variable on one page.

The clearest benefit of multivariate testing is that you get a better idea of which elements on your page are working or not working.

Should I use an A/B test or a multivariate test?

You might be asking yourself whether you should use an A/B test or a multivariate test. Good question. The answer? It depends.

If you want quick, meaningful results then an A/B test will be your best bet. A/B tests are very effective in telling you which out of two variables you should go with.

Now if you have a lot of traffic (a substantial amount of traffic is necessary for a multivariate test to be effective) to your website a multivariate test might provide more useful because of the sort of analysis it can provide.

In general, an A/B test is for big changes and multivariate testing is used for optimizing smaller elements.

Ever wanted to know how to do website usability testing?

When to use a multivariate test?

As mentioned, multivariate tests are good for testing smaller elements on a page. Other common uses of multivariate testing include:

• Testing both text and imagery on a website
• Testing both the text and color of CTA buttons
• Testing the number of input fields and CTAs together

What are the drawbacks of multivariate testing?

Multivariate testing is complex. It’s best performed by someone who has a good amount of knowledge on testing methodologies.

There is the issue of visitor traffic to consider when it comes to multivariate testing.

With an A/B test, you can simply do a 50/50 split to test something but when it comes to multivariate testing you need to split your traffic down into quarters, sixths, eighths and possibly more. This can be difficult to achieve without significant traffic.

A/B and multivariate testing resources and tools

There are a plethora of resources online which you can use to get started with A/B and multivariate testing as well as improve your testing process. To round off this article, here are some of our favorites.

Tools to do A/B and multivariate testing:

Optimizely — Customer experience optimization platform.

Crazy Egg — Heat maps can tell you where your users are hovering with their cursor.

Google Analytics — Web analytics for deep insights.

Adobe Target — Test combinations in real time and find the winning answer.

Kissmetrics — Behavior and engagement analytics platform.

UserVoice — Get feedback to influence product roadmaps.

Kampyle — Voice of customer platform.

HotJar — Heatmaps, visitor recordings and conversion funnels.

Zopim — Live chat to engage with customers.

Guides and handbooks for A/B and multivariate testing:

VWO Ebooks — An optimization website that has a ton of useful guides and ebooks all about testing and optimizing web and mobile experiences.

HubSpot A/B testing checklist — HubSpot’s essential A/B testing checklist.

A/B Testing Statistics: A Guide for Non-Mathematicians — A comprehensive run down on A/B testing and statistics for beginners.

Beginner’s guide to A/B testing by Kissmetrics — A walkthrough by the experts on testing.

Conclusion

Optimizing designs and boosting the user experience of websites and mobile apps is our bread and butter. A/B tests and multivariate testing are just two ways to help us achieve that goal; both are robust methodologies that can give us the power to create better designs.

When used properly, they can give you valuable insights that can lead to building better relationships with your users.