A/B Testing 101

 

Thinking about making a change to your website or app?  But will the change be better or worse? What do you do?  Determining if a change in software is good or bad is done as a controlled experiment called A/B testing.  Our guest writer, Leona Henryson is a seasoned blogger and UX designer and discusses A/B testing in this article.

A beginner’s guide to A/B testing

To be at your most effective, you need to know whether a change you’ve made is actually beneficial or not. Now you can simply guess, assume, or judge based on your own observations. All of those choices come with problems, however. And that is that they’re not very accurate.

A much better way to know what’s actually going on is by comparing what you had with what you have and seeing how the numbers compare. This will give you a far more unbiased and honest idea of what’s going on, which will then let you make far better-informed decisions about how to proceed.

Tests that let you work in this way are like turning on the lights in the darkness. Suddenly you can understand the landscape and stop stumbling about without knowing where you’re trying to go.

 

A/B testing explained

A/B testing is such a test. It allows you to pit two different versions of something (version A and B) against each other and see which comes out better. The basic idea is pretty straight forward. You randomly create two groups and then assign them to the two versions. Then you see how they react differently. As long as the groups are equal, whatever differences you find should be down to the difference between the two tests.

That’s why that random assignment is important. It makes certain that there are no institutionalized differences between the two groups. If, instead, you were to assign the first fifty people that come to one group and the next fifty to the next, you might have a time effect, where people who come to your test earlier are different (maybe they’re morning people?) than those who do so later.

By assigning them randomly you allow for everybody to have an equal chance to be assigned to either condition and – as long as the groups are large enough – that means differences should average out.

What is also hugely important is that you can actually measure the results that you’re looking for. There has to be some objective way to see which of the two tests triumphs. That’s why A/B tests are so often used online, as it’s incredibly easy to track a huge number of statistics, from how long they stick around to who goes through to buy your product.

Of course, A/B tests don’t have to be performed online. You could also run them in your neighborhood by, say, giving out two ads with different discount codes attached. Then the number of times each of the codes was used would give you some indication of which ad worked better.

 

Examples of A/B tests

There are many different versions of A/B tests. You can use them in a newsletter, for example. In that case you send one newsletter to one half of your audience and a different version to another. You can also run two different advertisements and see which one works better by seeing what the click through rate of the two different ads is.

Then there is the variation of the website, where people who click through to your website are shown two different versions of a page. Then you can see which page does better, in terms of how many people proceed down your sales funnel and how many people decide to head elsewhere instead.

 

The right way to do an A/B test

The trick to running an A/B test that actually tells you something useful is to only change one thing. This is important, because if you change several things then you won’t be able to track back which of the changes actually led to the increase that you’re looking for.

For example, say that you change your home page, by making bigger buttons, writing a high-quality copy and changing the colors as well. Then you run the test and find that there really isn’t any change between the two pages. You might wrongfully assume that the changes you made have had no effect. But that does not necessarily follow. What’s also possible is that two of the changes you made are pulling your results in opposite directions.

Even if you make several changes and the results improve, you’ve lost out. Why? Because again you don’t know which of the changes was actually beneficial. It’s entirely possible that if you would only have made two of the three changes the results would have been better with the last one being a small but noticeable drag on the other two.

Even worse, if you then want to go to the next page and use what you’ve learned from the first test and apply it here, you can’t. After all, you didn’t really learn anything and the only thing you can do is apply the changes wholesale.

A much better solution is to run an A/B/C test where you have your original version and then make two different versions where you vary two different things respectively from the original. Say in the B version you have bigger buttons and on version C you have the different text.

When you do that, you can compare the two different versions to the original A version and see how they do.

 

Things to consider when doing an A/B test

The first thing you’ll want to pay attention to is that you don’t irritate Google with your test. That’s a lot easier to do than you might realize. Google has some clear guidelines that you need to keep track of.

• The first is to make sure that the redirect is applied indiscriminately both to people and to search bots.
• Second, make sure you use a 301 redirect and not a 302 redirect. That’s the difference between a temporary and a permanent redirect.
• Finally, make sure that you run the test and then change it to the outcome you desire. If you leave the test running, Google will not be happy. Neither will you as you’ll probably get demoted for duplicate content.

Make use of hypotheses. Don’t go about indiscriminately changing things on your site in the hope of making things better. Most changes won’t have any effect. For that reason, your best bet is to read up on what other people have done and try the same things on your own site. You can also construct your own hypotheses. Perhaps you want to change the background color of your website to red because you think your visitors are adventurous and adventurous people like the color red.

Use the right analytical software. There are a lot of different software packages out there which will let you run A/B tests. And they’re probably pretty useful. I’ve never used them myself. Why? Because if you know what you’re doing you can already run most such tests using Google Analytics. Yes, even if you want to run a social media A/B test, you can use that program. All you need to do is add parameters to your URL which will show you where traffic is linking from.

 

So why not just run a time test?

So that’s A/B testing. As you can see, it isn’t that hard. Nonetheless, it might be harder than you’re willing to try – especially considering you can just change the page you’re using and then compare the before and after, right?

You can do that. The only problem, however, is that if you do then you’re introducing a huge amount of noise into your test. After all, when you compare the behavior of people visited your site this week to those who did so last week, you have no control over where those people are coming from. Perhaps in that time somebody links to your site from a blog post and sends their audience your way. If that audience is any different from who normally visits your site (maybe they stick around longer because the blogger vouched for you), that might well let you draw the wrong conclusions.

A/B testing avoids problems like that. It’s a far cleaner experimental design that lets you draw far cleaner observations. And that is well worth the extra time you need to invest in setting up such a test.