Chapter 30 Exercises: Field Experiments in Politics

Conceptual Exercises

Exercise 30.1 — The Causal Inference Problem A campaign manager tells you: "We know our canvassing program works — the voters we canvassed turned out at 65%, and the ones we didn't canvass only turned out at 48%. That's a 17-point effect." Explain, in terms a campaign manager would understand, why this 17-point difference is not a valid estimate of the canvassing effect. What additional information would you need to estimate the true effect? What research design would give you the most credible estimate?

Exercise 30.2 — ITT vs. LATE A GOTV canvassing experiment assigns 12,000 voters to treatment and 8,000 to control. After the election, you find that the treatment group turned out at 54.8% and the control group at 52.3% (a 2.5-point ITT effect). You also know that canvassers successfully contacted 38% of the treatment-group voters (the rest weren't home, refused to engage, or weren't found).

(a) Calculate the LATE estimate (the effect on actually-contacted voters). (b) Interpret the ITT and LATE in plain language: what do each of these numbers mean, and for which decisions is each most useful? (c) If the campaign wants to know "should we run this canvassing program again?", which estimate should they use? Why?

Exercise 30.3 — Power Analysis Intuition You want to run a GOTV canvassing experiment to detect an effect of 2 percentage points, with baseline turnout around 48% and a two-sided significance threshold of 5%. Without doing the full calculation, reason through the following:

(a) If you double the sample size, what happens to the minimum detectable effect? (b) If the true effect is only 1 percentage point instead of 2, what happens to the statistical power of a study designed to detect a 2-point effect? (c) If the contact rate is 35% instead of 50%, how does this affect what you need in terms of treatment-group assignment size to achieve 6,500 actual contacts?

Exercise 30.4 — Spillover in Experimental Design You are designing a canvassing experiment in a dense urban neighborhood with many multi-family apartment buildings. You plan to randomize at the individual voter level.

(a) Describe two specific spillover pathways that are likely in this setting. (b) What change to the randomization design would reduce spillover? What cost does this change impose on the experiment's statistical properties? (c) If you can't change the randomization unit, propose one design element that would allow you to measure (rather than eliminate) the spillover.

Exercise 30.5 — Blocked Randomization Design You are designing a randomized GOTV experiment with the following population: 40% of eligible voters voted in all three previous elections; 35% voted in two of three; 25% voted in one or zero. You believe canvassing effects may differ across these groups — specifically, that effects are larger for the lowest-propensity voters.

(a) Design a blocked randomization scheme that ensures balance on turnout history. (b) How does this blocked design affect your ability to analyze subgroup effects (the effect specifically for each turnout-history category)? (c) What statistical model would you use to analyze the results from your blocked design?

Applied Exercises

Exercise 30.6 — Evaluating the Social Pressure Mailer The Gerber, Green, and Larimer (2008) social pressure mailer experiment found turnout effects of approximately 8 percentage points. A campaign is considering running a social pressure mailer to 100,000 voters in its GOTV universe.

(a) Calculate the expected number of additional votes if the effect is 8 percentage points. (b) If a piece of direct mail costs $0.60 per piece to produce and mail, calculate the cost per additional vote generated. (c) Compare this cost-per-vote to a canvassing program with a 2.5-point per-contact effect and a cost of $12 per completed canvass contact. (d) Given the ethical concerns about social pressure mail discussed in the chapter, at what magnitude of backlash (measured in reduced enthusiasm from recipients, negative press, or other costs) would the cost-per-vote advantage disappear?

Exercise 30.7 — Regression Discontinuity Application You are studying the effect of winning a close election on a legislator's subsequent policy effectiveness. You have data on all state legislative races from the past decade, with vote margins and various outcome measures (bills passed, amendments adopted, legislative ratings).

(a) Explain how you would use a regression discontinuity design to estimate the effect of winning versus losing on legislative effectiveness. (b) What is the "running variable" in this design? What is the threshold? (c) Why is the comparison of legislators who barely won (winning margin 0–2%) to legislators who barely lost (losing margin 0–2%) credible as a causal estimate? (d) What threat to validity would exist if campaigns with better-funded incumbents systematically won more often in close races?

Exercise 30.8 — Difference-in-Differences Design A campaign ran a canvassing program in precincts A, B, and C but not in precincts D, E, and F (which it left as a comparison group). Turnout data from the current election and the immediately prior comparable election is below:

(a) Calculate the difference-in-differences estimate of the canvassing effect. (b) State the parallel trends assumption in plain language for this specific example. (c) The canvassed precincts are in the urban core; the control precincts are in the inner suburbs. Does this affect your confidence in the parallel trends assumption? Why? (d) Propose one piece of evidence you would gather to assess whether the parallel trends assumption holds.

Discussion Exercises

Exercise 30.9 — The Ethics of Political Experiments A researcher proposes running an informational field experiment in which some voters randomly receive information about an incumbent's real voting record (which the incumbent has not publicized) while others receive standard campaign mail. The information is accurate and publicly available but not widely known.

Discuss: (a) Is IRB consent required for this experiment? Should it be? (b) Is the researcher's use of accurate but electorally damaging information ethical? (c) Who should have access to the findings — the academic community, the campaigning organization that funded the study, or both? (d) How would your answers change if the information were about the incumbent's opponent rather than the incumbent?

Exercise 30.10 — Translating Evidence into Practice The chapter describes a gap between experimental findings and campaign practice: effect estimates from published studies are often applied to new contexts without adequate attention to whether they generalize. In a group discussion or written reflection:

(a) What specific contextual factors most affect the generalizability of a GOTV canvassing experiment's findings? (b) What would it take for a campaign to build its own context-specific evidence base rather than relying on published studies? (c) Is the Analyst Institute model — a research consortium that accumulates evidence across many campaigns — a good solution to the generalizability problem? What are its limitations?

Research Exercise

Exercise 30.11 — Literature Synthesis Find and read the original Gerber and Green (2000) article "The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment" in the American Political Science Review. Then find the Green, Gerber, and Nickerson (2003) replication study and the Arceneaux and Nickerson (2009) analysis of heterogeneous effects.

Write a 600–800 word synthesis that addresses: (a) What did the original study find, and what were its key innovations? (b) What did the replication and heterogeneity analyses add? (c) Based on these three articles, what is the state of knowledge about canvassing effects as of their publication dates? (d) What questions about canvassing effects did these studies leave unanswered?

Quantitative Exercise

Exercise 30.12 — Full Experimental Analysis You have results from a canvassing experiment with the following data:

• Treatment group: 8,500 voters assigned, 3,400 successfully contacted
• Control group: 5,000 voters assigned, 0 contacts
• Treatment group turnout: 4,505/8,500 = 53.0%
• Control group turnout: 2,575/5,000 = 51.5%

(a) Calculate the ITT estimate. (b) Calculate the contact rate (compliance rate). (c) Calculate the LATE estimate. (d) If the standard error of the ITT estimate is 0.009 (0.9 percentage points), calculate the 95% confidence interval for the ITT. (e) Is the ITT statistically significant at a 5% significance level? Explain how you determined this. (f) Interpret all four results (ITT, contact rate, LATE, and statistical significance) in plain language for a campaign manager deciding whether to expand the canvassing program next cycle.