A/B Testing in Marketing: How to Come Up with Good Hypothesis

In today's data-driven age of marketing, people often want to test and validate EVERYTHING with data. Data is important for every app marketing agency and in marketing in general, we can't go based on gut feel when all the evidence points in another direction

However, the majority fails to realize the other side of testing and collecting performance data to be used in marketing decisions. A smart performance agency understands that tests will cost time, money, resources and sometimes can even point in the wrong direction when the basis for a test is a poor hypothesis. 

Another big challenge in performance marketing is alignment around goals, metrics, and how to ‘prove’ what will market your product or service best. Understanding your goals and setting your expectations of a test is important and without it, you will be much like a person who is camping in flip-flops. Ouch!

Let's define "Hypothesis"

A hypothesis in the world of A/B testing is the prediction you create before running an experiment. It states clearly what is being changed, what you believe the outcome will be, and why you think that’s the case. 

Running the experiment will either prove or disprove your hypothesis and If the hypotheses are correct, then we can claim that we know that changing X will improve conversion rate, and we can implement that change for every visitor.

Your hypothesis should be more than a question (for instance, “Do images increase conversion rate?”). A hypothesis must also be an educated guess about the relationship between the changes you’ve made and the outcome of your test

But what makes a quality hypothesis?

A hypothesis is not a guess or hunch. It is not, “I think our landing page would improve conversion rates if we changed the layout” or “I believe this email would do better if I used blue instead of green.” 

A hypothesis needs to be more structured and calculated, with specifications that are able to be measured, analyzed, and evaluated. A good hypothesis has three parts: if…..then….rationale.

The IF part is very tricky and the majority mistakes this part and changes more than one variable in the same test. However, you need to isolate a single variable in a single A/B test, and multivariate only allowed when they are so related and their effect is almost similar.

The THEN part is when you predict the outcome of your hypothesis, the results. Your expectations should be relevant and within the boundaries of the test. This could be an increase in your conversion rate or a decrease in your cost per lead… you name it.

Here, you need to incorporate your existing data to be able to reasonably set your expectations and accurately predict the outcome of your experiment. Your current data will help you set your benchmarks to measure against.

The RATIONALE part is when you demonstrate that you have informed your hypothesis with research. It simply shows that you have done your homework.

Here is an example to make it easier. The rationale for testing a new landing page design might be changing color or taking your call to action to the top of the page to improve conversions because you found out after analyzing your heat map that 60% of your visitors don't go below section 2 of your landing page while your call to action is at the bottom of it

How to come up with a strong hypothesis?

Strong hypotheses are the foundation of all successful A/B tests. They ensure that you're testing the right thing, with the right people, at the right time. It's not an exact science. It's more like science fiction, where the laws of physics are constantly changing. 

The first step in coming up with a strong hypothesis is to identify your key metrics. A key metric is your KPI that you gauge very often and is the culmination of your efforts. So if you’re a growing company looking to improve your performance marketing, you’ll want to ensure that you have a good handle on your key marketing metrics. 

A key metric could be conversion rate, cost per lead, cost per signup, ROAS, ROI, or any other metric you usually use to indicate the success of your marketing activities. If you don't already have these clearly defined, ask customers or employees which metrics matter most.

Once you've identified the key metrics that matter to your brand, it's time to generate hypotheses. Come up with as many as you can. Let's say your key metric is engagement: What happens when we make our ads funny? What happens when we make them timely and relevant for the audience (e.g., around a holiday)? 

What happens when we use images of puppies? Now, what matters here is not whether you think these hypotheses will work — only that they lead to something else that can be tested and proved or disproved. That's why I like to think of these as "opportunities" that may (or may not) impact your metrics; who knows!

Once you have these metrics identified, you can start filtering and prioritizing them. For example, if the most important metric for your business is monthly recurring revenue (MRR), then go through your list and, for every item, ask yourself; Will this result in a better MRR? How? Why? 

Next, make a plan. Write down your potential hypotheses, define your key metrics, and write down the steps you'll need to take to test each hypothesis. If you're running an A/B test, write down which variations you're testing, pick a winner, and explain the reasons.

Finally, test your hypothesis. A hypothesis is only as good as your test, so it's important to be able to measure whether you're right or wrong. Keep in mind that a test is valid when it runs enough to generate enough data for decision-making.

 

That's it for this one, see you in the next article. Ciao!