Focus Groups for Ad Creative: What Agencies Get Wrong

Agencies moderating focus groups for ad creative is a practice that sounds rigorous but frequently produces the opposite of useful. When the people who made the work are also running the room, the research stops being research and starts being a performance , one where the audience is managed toward a conclusion rather than heard.

That is not a cynical observation. It is a structural problem. The conflict of interest is baked in from the moment the moderator’s agency created the stimulus material being tested. Objectivity is not a personality trait you can will into existence when your work is on the table.

This article is about what that problem looks like in practice, why it persists, and what a more commercially honest approach to creative research actually requires.

Why the Conflict of Interest Is Structural, Not Personal

Most agency people who moderate focus groups on their own creative work are not consciously trying to manipulate the outcome. They believe they can hold the tension between advocacy and objectivity. Some are genuinely skilled moderators. None of that resolves the problem.

The issue is structural. The agency has already committed creative resources to the work. There is a relationship with the client to protect. There are internal champions who believe in the campaign. By the time a focus group is convened, the work has usually cleared multiple internal reviews, been presented to senior clients, and generated real organisational momentum. The research is rarely positioned as a genuine decision point. It is positioned as a final check.

In that context, a moderator from the same agency is not a neutral facilitator. They are someone with skin in the game sitting in a room where the game is being played. Even unconsciously, they will probe harder on positive responses, let negative comments pass without follow-up, and frame questions in ways that prime participants toward charitable interpretations.

I have sat in on debrief sessions where the agency moderator’s summary bore only a passing resemblance to what I had observed through the one-way mirror. Not through dishonesty, but through the entirely human tendency to hear what you are hoping to hear. The edit happens before the report is written.

If you are interested in the broader landscape of research methodology and how it feeds into marketing strategy, the Market Research and Competitive Intel hub covers the full range of approaches , from qualitative to quantitative, and how each fits into a planning cycle.

What Good Focus Group Moderation for Creative Actually Requires

Running a focus group on ad creative is a specific skill. It is not the same as running a focus group on product satisfaction or brand perception, and agencies that treat it as a generic qualitative technique tend to produce findings that are either useless or actively misleading.

The moderator needs to understand how people process advertising. That means understanding that first reactions are not the same as considered responses. It means knowing that participants in a research setting will often critique creative work on rational grounds, because that is what the context demands, even when their actual response to the same ad in the real world would be emotional and largely unconscious. It means being able to distinguish between a participant who genuinely dislikes a piece of work and one who is performing dislike because the group dynamic has moved in that direction.

None of this is impossible. There are excellent independent research agencies that specialise in exactly this kind of work. The problem is that commissioning them requires the client to insist on separation, and many clients do not know to ask for it. They assume that the agency running the research is the appropriate party to run the research. That assumption is almost always wrong when the agency also made the work.

The practical requirements for effective creative focus group moderation are not complicated. The moderator should have no prior involvement with the creative development. The discussion guide should be written or reviewed by someone independent of the agency. Stimulus material should be presented in context, not in isolation, because advertising is not experienced in isolation. And the debrief should include verbatim participant quotes, not just the moderator’s thematic summary.

The Stimulus Problem: How Creative Gets Tested in Ways That Guarantee Distortion

Even setting aside the moderator conflict, the way stimulus material is typically presented in agency-run focus groups introduces systematic distortion.

Animatics and rough cuts are not ads. They are approximations of ads, and participants know it. When you show someone a storyboard or a rough edit and ask them to respond as if it were a finished piece of work, you are asking them to perform an imaginative leap that most people cannot make reliably. The gap between an animatic and a finished thirty-second television spot is not just production quality. It is pacing, music, performance, colour grading, and the hundred small decisions that determine whether something feels confident or uncertain. Participants respond to what they see, not to what the finished work might be.

Agencies know this. They use rough stimulus anyway, because testing finished work is expensive and because the research is usually scheduled before production is complete. That is a legitimate commercial constraint. The problem comes when the findings are reported as if the stimulus limitation does not exist, and when negative responses to rough creative are attributed to the concept rather than the execution quality.

I spent a significant period of my career managing large media and creative budgets across multiple categories. The research that held up over time was almost always the research that was honest about what it could and could not tell you. The research that caused problems was the research that overstated its own certainty, usually because someone needed a green light and the focus group was the mechanism for providing it.

Understanding how users actually respond to content in context, rather than in a research room, is a challenge that applies well beyond focus groups. Tools like Hotjar’s issue-spotting features offer a different lens on real-world behaviour, one that does not depend on participants articulating their responses in a group setting.

Group Dynamics and the Problem of Social Desirability

Focus groups have a well-documented structural weakness that applies regardless of who moderates them: people behave differently in groups than they do alone. They moderate their responses toward what they perceive to be socially acceptable. They are influenced by dominant voices. They are reluctant to sustain a position that the group has moved away from, even if they privately hold it.

For ad creative testing, this creates specific problems. Advertising often works through mechanisms that participants would find difficult to admit to in a group setting. Aspiration, social signalling, status, and anxiety are all powerful advertising levers, but they are not things people readily acknowledge as influencing them. A participant who finds a luxury brand campaign genuinely appealing because it flatters their self-image is unlikely to say so in front of seven strangers. They will find a more rational objection instead.

This is not a reason to abandon focus groups. It is a reason to be precise about what they can and cannot tell you. Focus groups are useful for identifying comprehension problems. They are useful for surfacing strong negative reactions that might indicate a genuine misfire. They are less useful for predicting whether a campaign will perform, and they are almost useless for predicting the emotional or behavioural response of a mass audience based on the responses of eight to twelve recruited participants.

When I was judging at the Effie Awards, the work that had been focus-grouped into safety was often the work that struggled most to demonstrate real effectiveness. The campaigns that showed genuine commercial impact were frequently the ones where someone had backed a strong creative instinct against equivocal research, not because research is bad, but because they understood what the research was actually measuring.

What Clients Should Actually Ask Before Commissioning Creative Research

If you are a client commissioning focus groups on ad creative, there are five questions worth asking before the research begins. They are not complicated, but they are rarely asked.

First: who is moderating, and what is their relationship to the creative work? If the answer is that the moderator works for the agency that produced the stimulus, the research design needs to change. That is not a negotiation point. It is a basic methodological requirement.

Second: what is the research actually trying to answer? “Does the creative work?” is not a researchable question. “Do participants understand the core message without prompting?” is. “Does the opening sequence hold attention?” is. Be specific about what a useful finding would look like, and what you would do differently based on different outcomes.

Third: how will the stimulus be presented, and what are the limitations of that format? If you are testing an animatic, the debrief should acknowledge explicitly that participant responses may be affected by production quality, and findings should be weighted accordingly.

Fourth: what is the decision this research is informing? If the decision has already been made and the research is a formality, say so. Running focus groups as a validation exercise rather than a genuine decision input is not necessarily wrong, but it should be acknowledged, because it changes how you should weight the findings.

Fifth: what will the debrief include? Ask for verbatim quotes alongside thematic summaries. Ask for a breakdown of responses by participant, not just aggregate impressions. Ask what the moderator heard that surprised them. The surprises are often where the useful information lives.

The way creative research fits into a broader planning process matters enormously. Effective market research should feed decisions, not decorate them. If you want to think more carefully about how research integrates into strategy, the Market Research and Competitive Intel hub is a useful place to work through the different methodologies and their appropriate applications.

When Agencies Should and Should Not Be Involved in Moderation

There is a version of agency involvement in creative research that is entirely legitimate. Agencies can and should be involved in defining the research objectives, reviewing the discussion guide, and observing sessions from behind the glass. They should be part of the debrief conversation, because they have creative context that an independent research agency may lack.

What they should not do is run the room. The moderator’s job is to create conditions where participants feel safe saying what they actually think, including things that are critical of the work. That requires a degree of separation from the work that the agency cannot credibly claim.

There is also a category of research where agency moderation is less problematic: exploratory research conducted before creative development begins. When an agency is running focus groups to understand audience attitudes, category perceptions, or unmet needs, the conflict of interest is much lower because there is no existing work to protect. The problem is specific to creative evaluation, where the agency has already produced something and is now asking consumers to assess it.

The industry has moved toward more sophisticated testing approaches over time. Quantitative creative testing platforms that measure emotional response, attention, and recall without the social dynamics of a group setting have become more accessible. These are not replacements for qualitative research, but they offer a useful complement, particularly for comparing multiple creative routes against each other without the distortion that comes from group discussion.

Optimizely’s work on experimentation and testing methodology reflects a broader shift toward evidence-based creative decisions that does not rely solely on what participants say in a room. The direction of travel in the industry is toward triangulating multiple signals rather than treating any single research method as definitive.

The Broader Problem: Research Used to Manage Risk Rather Than Inform Decisions

The deepest problem with how agencies use focus groups on ad creative is not methodological. It is organisational. In many cases, the research is not genuinely intended to inform a decision. It is intended to provide cover for a decision that has already been made, or to share accountability in case the campaign does not perform.

This is not unique to marketing. It is a pattern that shows up in most large organisations: research is commissioned not because the findings will change anything, but because having conducted research provides a defensible position. “We tested it and consumers responded positively” is a sentence that protects careers, even when the testing was designed in a way that made a positive finding almost inevitable.

The waste here is not just financial, though running focus groups is not cheap. The waste is the opportunity cost of research that could have been genuinely useful. Time spent conducting research designed to validate rather than inform is time not spent on research that might actually change something.

I think about this the same way I think about the industry conversation around sustainability in advertising. There is a lot of focus on the carbon impact of ad serving, which is measurable and therefore easy to report on. There is much less focus on the strategic waste, the bad briefs, the misaligned campaigns, the research that confirms what everyone already believed. The visible costs get managed. The invisible costs accumulate.

Genuine creative research, designed to surface real consumer responses rather than to validate existing work, requires a willingness to hear things you do not want to hear. That requires structural independence in the moderation, honest stimulus presentation, and a client-agency relationship where negative findings can be discussed without the conversation becoming adversarial. Those conditions are not complicated to create. They are just rarely prioritised.

Thinking about how emotional response shapes consumer behaviour, and how to design research that captures it honestly, is a challenge that extends well beyond focus groups. The principles behind designing for emotional needs apply equally to creative testing: you need to understand what people feel, not just what they say they feel.

The BCG perspective on lean methodology and eliminating structural waste is a useful frame for thinking about research design too. The question is not whether you are conducting research. The question is whether the research is structured in a way that could actually change the outcome, or whether it is process overhead that adds cost without adding insight.

About the Author

Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.

Frequently Asked Questions