Control Group Marketing: Are You Measuring What Worked?

Control group marketing is the practice of deliberately withholding marketing activity from a defined segment of your audience to establish a genuine baseline, so you can measure what your campaigns actually caused rather than what they coincided with. Without a control group, almost every attribution model you rely on is telling you a story, not a fact.

Most marketers are measuring correlation and calling it causation. Control groups are the structural fix for that problem.

Why Most Campaign Measurement Is Quietly Broken

Early in my career, I was deeply invested in lower-funnel performance metrics. Click-through rates, cost per acquisition, return on ad spend. The numbers looked compelling. Clients were happy. The dashboards were full of green arrows. It took me longer than I would like to admit to recognise that a material portion of those conversions were happening anyway, and that we were essentially taking credit for consumer intent that existed before we showed a single ad.

This is the core problem with standard last-click and even multi-touch attribution. It assigns credit to marketing touchpoints based on proximity to conversion, not based on whether those touchpoints caused the conversion. Someone who searched for your brand name, clicked a retargeting ad, and then purchased was probably going to purchase regardless. The retargeting ad did not create that intent. It just happened to be in the room when the decision was made.

Control group marketing solves this by creating a structured comparison. You take a representative sample of your audience, withhold the campaign from that group, and then compare their conversion behaviour against the group that received the campaign. The difference between the two groups is your incrementality, the actual lift your marketing created. Everything else is noise dressed up as signal.

If you are thinking carefully about how your go-to-market activity connects to genuine business growth, rather than just activity metrics, the broader principles behind control group testing fit squarely into that discipline. The Go-To-Market and Growth Strategy hub on The Marketing Juice covers the commercial thinking that sits around measurement decisions like this one.

What a Control Group Actually Looks Like in Practice

The mechanics are straightforward, even if the execution requires discipline. You define your target audience. You randomly split that audience into two groups: the test group that receives your campaign, and the holdout group that does not. You run the campaign. You compare conversion rates, revenue per user, or whatever outcome metric matters. The gap between the two groups, adjusted for any pre-existing differences, is your incremental lift.

The word “randomly” is doing a lot of work in that description. A holdout group that is not genuinely randomised will produce results you cannot trust. If your holdout group accidentally skews toward users who were already less likely to convert, your lift numbers will look artificially strong. If it skews toward high-intent users, your lift will look weaker than it is. The randomisation is not a technicality. It is the entire foundation of the test.

Most programmatic platforms now support holdout testing natively. Google’s conversion lift studies, Meta’s conversion lift tool, and third-party measurement platforms like Measured or Northbeam all offer versions of this. The infrastructure exists. The barrier is not technical. It is cultural. Marketers and their clients are often reluctant to withhold spend from any segment of a paying audience, because the short-term optics of “not marketing to someone” feel uncomfortable. That discomfort is worth pushing through.

The Retargeting Problem Nobody Talks About Loudly Enough

Retargeting is the category where control group testing tends to produce the most uncomfortable results. When I was running agencies and managing large performance budgets, retargeting consistently delivered the best reported ROAS of any channel. It looked like the engine of the operation. Then we started running holdout tests on retargeting campaigns for several clients, and the picture changed significantly.

In most cases, a substantial portion of the conversions attributed to retargeting came from users who would have converted without the retargeting exposure. These were people who had already decided to buy. They had added items to a cart, visited a pricing page, or searched for the brand directly. The retargeting ad intercepted them on the path to a decision they had already made. The platform reported a conversion. The dashboard showed strong ROAS. The budget renewed. And the actual incremental contribution of that spend was a fraction of what the numbers suggested.

This is not a reason to abandon retargeting. It is a reason to measure it properly. Some retargeting activity genuinely re-engages users who had drifted away from a purchase decision. Some of it is pure waste. Without a holdout group, you cannot tell the difference. You are managing a budget based on a story the attribution model is telling you, not based on what is actually happening.

The broader point here connects to something I have believed for a long time: most performance marketing is better at capturing existing demand than creating new demand. That is useful, but it is not the same as growth. Real growth requires reaching people who were not already looking for you. That is a harder problem, and it is one that market penetration strategy frameworks address more honestly than most performance dashboards do.

How to Design a Control Group Test That Produces Usable Data

There are five things that determine whether a control group test is worth running.

Audience size. You need enough people in both groups to reach statistical significance. The exact number depends on your expected conversion rate and the lift you are trying to detect, but as a rough orientation, you are rarely working with reliable data if either group has fewer than a few thousand users. Smaller audiences produce wide confidence intervals, and wide confidence intervals mean you cannot draw conclusions with any confidence.

Clean randomisation. As noted above, this is non-negotiable. Use platform-native tools or a third-party solution that handles randomisation properly. Do not manually segment your holdout group based on geography, recency, or any other variable that could correlate with conversion intent.

Holdout purity. The holdout group must not receive the campaign through any channel. If you are running a holdout on a paid social campaign but the holdout group is still receiving email campaigns, search ads, or direct mail for the same product, your test is contaminated. You are measuring the incremental effect of paid social on top of everything else, which may or may not be what you want to know.

Test duration. Run the test long enough to capture a full purchase cycle. If your average customer takes three weeks from first consideration to purchase, a one-week test will undercount conversions in both groups and distort your lift calculation. A common mistake is cutting tests short because the early numbers look good or bad. Let the test run.

Pre-test baseline. Before you run the test, confirm that your test and holdout groups are comparable on the metrics that matter: historical conversion rates, average order value, session frequency. If the groups are not comparable before the campaign starts, any difference you observe during the campaign is uninterpretable.

Geo-Based Holdouts: A Practical Alternative

User-level holdout tests are the gold standard, but they are not always feasible. Some channels do not support user-level holdouts. Some businesses have audience sizes too small to split reliably. In these cases, geo-based holdout testing offers a workable alternative.

The principle is the same, but instead of withholding the campaign from a random sample of users, you withhold it from a defined geographic region. You run the campaign in your test markets and withhold it from your holdout markets. You then compare conversion trends across the two sets of markets, controlling for any pre-existing differences in baseline performance.

Geo-based tests are particularly useful for TV, out-of-home, and radio, where user-level attribution is impossible. They are also useful for testing brand campaigns, where the effect on conversion may be diffuse and delayed rather than immediate. The limitation is that geographic markets are never perfectly comparable. There are always confounding variables: local economic conditions, competitor activity, seasonal patterns that differ by region. You need to account for these in your analysis, or your lift numbers will be unreliable.

I have run geo-based holdouts for national retail clients where the results fundamentally changed how we allocated the media budget. In one case, a TV campaign that had been running for two years and was credited with strong brand lift in post-campaign surveys showed almost no measurable incremental effect on sales in the holdout markets. That was a difficult conversation to have. It was also the most commercially useful conversation that client had about their media investment in years.

What Incrementality Testing Reveals About Budget Allocation

The real value of control group marketing is not the individual test result. It is the cumulative picture that emerges when you run holdout tests systematically across your channel mix over time. That picture tends to challenge assumptions that have been baked into budget allocations for years.

Channels that look expensive on a cost-per-acquisition basis often show strong incremental lift when tested properly, because they are genuinely creating demand rather than just capturing it. Channels that look efficient on standard attribution metrics often show weak incremental lift, because they are predominantly intercepting users who were already converting. The budget implications of this are significant.

This connects to a broader point about commercial transformation in go-to-market strategy: the businesses that consistently outperform are not the ones with the largest marketing budgets. They are the ones who understand which parts of their budget are genuinely working and have the discipline to reallocate away from the parts that are not. That requires measurement infrastructure that most organisations do not have, and a willingness to act on uncomfortable findings that most organisations do not demonstrate.

When I was growing an agency from around 20 people to over 100, one of the things that differentiated our best client relationships was this kind of honest measurement conversation. Clients who were willing to run holdout tests, sit with the results, and make budget decisions based on incrementality rather than reported ROAS consistently grew faster than clients who optimised based on dashboard metrics alone. The measurement discipline was not separate from the growth strategy. It was part of it.

The Organisational Resistance You Will Need to Manage

Introducing control group testing into an organisation that has been running on standard attribution metrics is not purely a technical exercise. It is a change management exercise. The results of holdout tests frequently conflict with the reported performance of campaigns that have been celebrated internally, budgets that have been defended in boardrooms, and agency relationships that have been built around particular metrics. People have skin in the game, and incrementality testing can make that skin uncomfortable.

The most effective way to manage this is to frame holdout testing as a tool for making better investment decisions rather than as an audit of past decisions. You are not trying to prove that previous campaigns were ineffective. You are trying to build a more accurate model of what works, so that future budgets can be allocated more intelligently. That framing is more accurate and more productive.

It also helps to start with a channel or campaign where the results are likely to be positive. Running your first holdout test on a channel that everyone internally suspects might be underperforming, and then publishing results that confirm those suspicions, is a politically difficult way to introduce the methodology. Start somewhere where the incremental lift is likely to be real and visible. Build credibility for the measurement approach before you use it to challenge sacred cows.

There is also a conversation worth having about what happens to the holdout group. In most tests, the holdout group converts at a lower rate than the test group. That means you are accepting some short-term revenue loss in exchange for measurement accuracy. For most businesses, this is a reasonable trade. For businesses operating on very thin margins or in highly competitive acquisition environments, it requires more careful consideration. Understanding those trade-offs is part of why go-to-market execution feels harder than it used to, and why measurement rigour has become a competitive differentiator rather than just a nice-to-have.

Control Groups and Brand Marketing: A Different Challenge

Most of the discussion around control group testing focuses on direct response campaigns, where the conversion event is measurable and relatively proximate to the marketing exposure. Brand marketing presents a harder version of the same problem.

Brand campaigns are designed to shift awareness, perception, and long-term purchase propensity. The effects are real, but they are diffuse and delayed. A consumer who sees a brand campaign today may not convert for six months. Standard holdout tests, which typically run for a few weeks, will miss much of this effect. You need longer test windows, more sensitive outcome metrics, and a willingness to accept that some brand effects are genuinely difficult to isolate.

This does not mean brand marketing is unmeasurable. It means it requires different measurement approaches. Brand lift studies, long-term geo experiments, and econometric modelling can all contribute to a more complete picture. The mistake is applying direct response measurement frameworks to brand activity and concluding that brand marketing does not work because it does not show up in short-term conversion data. That is a measurement failure, not a marketing failure.

I have judged the Effie Awards, which are specifically designed to recognise marketing effectiveness. One of the consistent patterns in the most awarded work is that the brands making the strongest effectiveness cases are not the ones with the cleanest attribution models. They are the ones with the most honest and multi-layered approach to measurement, combining short-term sales data with long-term brand tracking and, where possible, controlled experiments. That combination is harder to produce than a clean ROAS number, but it is considerably more credible.

If you are building a measurement framework that spans brand and performance activity, the thinking around go-to-market launch strategy is relevant here, particularly the emphasis on defining success metrics before you launch rather than retrofitting them to whatever the data shows afterwards.

The discipline of control group marketing is, at its core, the same discipline that distinguishes serious commercial thinking from marketing theatre. It asks a simple question: would this have happened anyway? Answering that question honestly, and then acting on the answer, is one of the most commercially valuable things a marketing team can do. It is also, in my experience, one of the rarest. Most of the growth strategy work that matters comes back to this kind of honest measurement, and you can find more thinking on that across the Go-To-Market and Growth Strategy section of The Marketing Juice.

About the Author

Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.

Frequently Asked Questions