What makes a marketing experiment successful? What makes it useful and ensures it provides valuable and actionable insights? It needs to be high quality.
Sounds simple, doesn't it? But with more tools being released to help marketers run experiments, and a greater need to prove return on investment (ROI) for marketing spend, there's never been a better time to get experimentation right.
We’ve identified four characteristics that'll turn your experiment into a high-quality experiment. And we spoke to two brands from the Nordics and Benelux that will show you why they run them — and how you can too.
1. Link questions to actions
In a nutshell: Only run experiments if you plan to take action on the results.
What to do: Before you start an experiment, think about the potential outcomes. If you are asking the question: "What is the cost per incremental conversion of my YouTube campaign?", consider the possible outcomes and what you will do once you have the results.
One trick you can employ is to create an action tree. This will help you:
- Identify your question
- Brainstorm potential outcomes
- Choose an action for each outcome
What others have done: TUI is one of the world’s biggest travel companies. It runs large-scale experiments to measure the incrementality of its travel campaigns.
Incrementality focuses on measuring the true causal impact of advertising spend. This is done by comparing the key business metric for a group of people who were exposed to advertising, with a group of people who were not.
“At TUI, there is a close connection between the marketing analytics team and media operations teams,” says Nicolas Elshout, digital marketing manager at TUI. “This ensures we set up experiments in a way that helps us answer the most important marketing questions and validate assumptions in a data driven way.”
Once the experiment is finalised and the results have been analysed, the teams work together on appropriate changes to reflect the new learnings. Elshout explains: “For instance we might adjust the target cost per acquisition of our ads so we drive growth at a higher profitability.”
2. Choose the right metric for the KPI
In a nutshell: Use important business metrics as the key performance indicators (KPIs) for your experiments, as long as they are statistically measurable.
What to do: First, make sure an experiment is evaluated on metrics that are critical to business goals. If your goal is to grow the user base of your newly designed app, then app installations would be a relevant KPI. If your goal is to decrease return rates of your sold products, the number of returned products would be your main metric. Always evaluate an experiment with the metric you are trying to impact.
Secondly, when running an experiment you make a comparison between two groups of people — for instance, those who have seen your ad vs. those who have not — and measure a statistically significant difference.
Statistical significance means that you can be confident that your marketing is driving the results of your experiment and this isn’t influenced by other factors or chance. But to understand this, you need a lot of data.
When choosing a metric, consider how often this metric or event happens. More frequent occurrences lead to more data which, in turn, will make it easier to get statistically significant results. A car dealership, for example, might not want to use car sales as the experiment KPI because it happens only once every seven years in a person's life. Instead, they might consider test drives booked or website interactions.
What others have done: Just Eat Takeaway evaluates all its advertising campaigns on metrics that are directly tied to business outcomes and success.
“We are a heavy user of incremental products across all marketing channels,” says Rico Stuijt, incrementality principal at Just Eat Takeaway.
The team runs incrementality tests across all their marketing channels (apps, search, display, and video) and evaluates the success on their two most important metrics: incremental conversions and incremental new customer conversions.
Stuijt continues: “We combine this information into one aggregated metric, including a lifetime component. This allows us to make cross-channel comparisons that are like-for-like and make the most accurate decisions to optimise our marketing investments.”
3. Consider the skill set required
In a nutshell: To set up experiments and make reliable recommendations based on the results, you need a strong understanding of statistics.
What to do: At this stage you need to consider your internal resources. Who will be running the experiments and what skills do they need?
You’ll either need to put in place a data science or analytics team, or educate yourself on the principles of experimentation, such as A/B testing and power analyses.
Without this specific skill set, you won’t get the most out of your experiments and you could even make suboptimal decisions.
What others have done: Once TUI had the right skills in place, they were able to run more regular experiments. “We only recently started conducting the incrementality test for Google Ads Search,” says Jeroen Maaijen, TUI’s digital performance marketing manager. “So far we have gained valuable insights and learnings, especially on how to optimise this process in the future. As we continuously look at the added value of our paid marketing channels we will embed incrementality testing in our regular way of working. My advise would be; just start using it, learn and fine-tune along the way”
TUI’s Nicolas Elshout, continues: “Randomised control tests involve opportunity costs due to potential missed sales. To minimise the risk of unsuccessful tests, we rely on statistics to navigate uncertainties”.
They are then able to use their inhouse analytics teams to come up with rigorous experiment plans for each test to validate its setup. “We perform power analyses before every experiment to determine its duration to achieve statistically significant results, ensuring reliable and actionable conclusions,” concludes Elshout.
4. Experimentation should be ‘always-on’ to adapt to change
In a nutshell: The world changes constantly and so does the effectiveness of your marketing efforts. That’s why you should test regularly and not generalise learnings.
What to do: This final step brings us full circle. You have to know the wider context of what’s going on when you set your business goals, define your metrics, and begin an experiment.
We often see companies doing incrementality tests in January and February and then generalise the learnings across the year. But high-quality experiments test broadly across different regions, across different marketing channels, and also over time.
A good trick here to help plan the different experiments throughout the coming quarters, is to build your company’s “test and learn programme”. This way you document the most important experimentation projects along with their objective, methodology, owner, etc.
Experimentation should be a part of your marketing process and always happening.
What others have done: Performance of your advertising campaigns can change significantly over time, geographical regions, and advertising channels. “That makes sense, as both external and internal factors have an impact on performance,” says Just Eat’s Rico Stuijt.
“Changes in the macroeconomic climate, user behaviour, and competitor activations are big factors. And, internally, we are continuously optimising our advertising campaigns with new creatives, targeting, and other optimisations that influence performance. Therefore, regular testing has been essential to make sure our performance estimates are timely and accurate.”
