Case Study 4.1: The Advertising Attribution Problem — What Moved the Numbers?

Background

It is the third week of October in the Garza-Whitfield Senate race. The Garza campaign has just completed a two-week burst of $1.2 million in television and digital advertising focused on Whitfield's record on prescription drug prices. The ads ran primarily in the two largest media markets. A tracking poll conducted at the end of the advertising period shows Garza's lead growing from 2 points to 6 points among likely voters, and her favorability among voters aged 55 and older — the demographic most concerned about drug prices — rising from 41% to 49%.

Campaign leadership is delighted. The communications director prepares a memo crediting the advertising buy for the shift in numbers, recommending that the campaign replicate the strategy in the Southwestern media market. Before the memo is finalized, Nadia Osei reviews the data and raises several concerns.

What Nadia Noticed

Nadia's first step was to pull together everything that happened during the two-week period — not just the advertising buy, but the full context.

National environment: A national tracking poll released mid-period showed Democrats gaining 1.8 points on the generic congressional ballot, likely driven by a Consumer Price Index report that showed higher-than-expected inflation. A significant share of the polling movement Garza experienced may simply be a reflection of a national Democratic wave forming.

Earned media: In week two of the advertising period, Whitfield gave an interview to a regional newspaper in which he declined to criticize a pharmaceutical industry donation to a Republican Super PAC. The story was picked up by the state's largest newspaper and generated significant local coverage. This independent event may have moved voters on the drug prices dimension independently of Garza's paid media.

Polling methodology: The baseline poll (2 points) was conducted by the campaign's internal polling firm, which tends to show the campaign's candidate performing slightly better than external polls. The follow-up tracking poll (6 points) was conducted by an independent firm as part of a media outlet's polling project. The apparent 4-point gain may partially reflect house effects — methodological differences between the two firms — rather than real movement.

Comparison market: The Southwestern media market, where no advertising ran during this period, also showed a 1.5-point improvement in Garza's numbers. If the advertising were the primary driver, you would expect to see larger gains in markets where advertising ran, but the comparison market narrows the unexplained gap to 2.5 points rather than 4 points.

The Attribution Problem

What Nadia is confronting is the fundamental attribution problem in political advertising research: campaigns observe that advertising runs and numbers improve, and they conclude that advertising caused the improvement. But in an observational context — where advertising is not randomly assigned — this conclusion faces severe confounding.

The true counterfactual — what would have happened to Garza's numbers during this period without the advertising — is unobservable. Nadia can approximate it using the comparison market (Southwestern), but she cannot be sure the markets are otherwise comparable. The national environment moved for everyone during this period. The Whitfield earned media story benefited Garza everywhere, not just in the advertising markets. The polling firms used at baseline and follow-up were different.

A back-of-envelope decomposition of the 4-point movement looks like this:

• National environment shift (estimated from Southwestern market): ~1.5 points
• Whitfield earned media self-injury (estimated from issue-specific favorability movement, non-advertising markets): ~0.8 points
• House effects between the two polling firms (estimated from historical difference between the two firms' readings on the same races): ~0.5 to 1.0 points
• Residual attributable to advertising: ~0 to 2.2 points

The uncomfortable truth is that after accounting for alternative explanations, the advertising's causal contribution to the polling movement is somewhere between negligible and 2.2 points — a range wide enough to be nearly useless for decision-making. The communication director's attribution of the full 4-point movement to the advertising is almost certainly wrong.

Jake Rourke's Reaction

When Jake Rourke, Whitfield's campaign manager, heard about Garza's polling surge, his reaction was characteristically intuitive. "They had a good week," he said. "But the drug pricing message doesn't move rural voters, it moves suburban seniors. And suburban seniors were already trending toward Garza. I'm not worried." This assessment is not wrong — the age and geographic breakdown of the movement is consistent with Jake's intuition — but it is also not backed by systematic analysis. Jake is right about the pattern but for insufficiently specified reasons, which means he may draw the wrong conclusion about how to respond.

Implications for the Communications Director's Memo

Nadia has to decide how to communicate her concerns. If she simply says "we don't know whether the advertising worked," she will be dismissed as unhelpfully academic. If she lets the overconfident memo stand, the campaign may make a $400,000 decision based on a flawed causal attribution.

Her approach: she revises the memo's framing without removing the advertising recommendation. The revised memo notes that polling movement during the period was likely multi-causal, identifies the national environment and earned media as contributing factors, and recommends the Southwestern market buy based not on the attribution of a 4-point gain but on the more defensible logic that drug pricing messaging has documented effectiveness with the Southwestern market's demographic composition — a claim rooted in experimental evidence from prior campaigns rather than in the potentially spurious correlation from this two-week window.

Discussion Questions

1. Identify all the confounds in the original attribution claim (advertising → polling gain). Which do you think accounts for the largest share of the observed movement, and why?

2. Nadia estimates that the advertising's causal contribution is between 0 and 2.2 points. How should this range of uncertainty affect the campaign's decision about the Southwestern market buy? Is this range decision-relevant?

3. The communications director's memo is advocating for a position (replicate the buy) that Nadia thinks may be based on a flawed premise. How should a campaign analyst handle this kind of tension between analytical integrity and organizational dynamics?

4. What experimental design would allow a future campaign to estimate the causal effect of issue-based advertising more rigorously? What are the practical obstacles to implementing such a design?

5. Jake Rourke's intuitive assessment ("drug pricing doesn't move rural voters") is directionally consistent with the data but not analytically grounded. Is this kind of expert intuition useful? How should it be weighted against systematic analysis?

Key Analytical Concepts Illustrated

• Attribution fallacy: assigning full causality to one factor in a multi-causal environment
• Confounding in observational analysis: the challenge of isolating advertising effects from concurrent events
• House effects: systematic differences between polling firms that affect apparent movement
• Comparison market design: using a geographic control to approximate counterfactual conditions
• Analytical diplomacy: communicating uncertainty without paralyzing organizational decision-making