Chapter 10 Exercises: Reading and Evaluating Polls

Section A: Conceptual Questions

Exercise 10.1 — The AAPOR Checklist

A campaign releases the following poll summary to the press:

"A new survey finds Senator Whitfield leading challenger Garza by 6 points among likely voters in the state. The poll of 800 respondents was conducted last week and has a margin of error of ±3.5 points."

Apply AAPOR's transparency checklist to this poll description. List every required disclosure that is missing or inadequate in this description, and for each gap, explain what information you would need and why its absence prevents you from fully evaluating the poll's credibility.

Exercise 10.2 — Population Definitions

A state has 4.2 million adults 18+. Of these: - 68% are registered to vote (2.86 million) - Of registered voters, 72% are expected to vote in an upcoming midterm election - The expected electorate is therefore approximately 2.06 million likely voters

Candidate A leads Candidate B by: - 8 points among all adults - 5 points among registered voters - 2 points among likely voters

a. Explain why these three estimates can all be "correct" simultaneously. b. If Candidate A is the Democratic candidate, what does the narrowing gap from adults to likely voters tell you about the composition of each population? c. A journalist publishes the headline "New Poll: Candidate A Leads by 8 Points." What is misleading about this headline for a likely-voter election? d. Under what conditions would the registered voter estimate be more useful than the likely voter estimate for predicting the election outcome?

Exercise 10.3 — Margin of Error Arithmetic

A poll of 750 likely voters shows: - Candidate A: 52% - Candidate B: 44% - Undecided: 4%

a. Calculate the margin of error for each candidate's individual share (at 95% confidence). b. Calculate the margin of error for the lead (A minus B). c. A reporter writes: "Candidate A leads by 8 points, though the lead falls within the margin of error of ±4%." Is this statement correct? Explain. d. Calculate the 90% confidence interval for the lead. e. A second poll of 1,500 likely voters shows A at 50%, B at 46%. Is the difference between these two polls' margins (8 points vs. 4 points) statistically significant? Show your work.

Exercise 10.4 — House Effects

You have access to 15 polls of a Senate race conducted within a 30-day period. After calculating concurrent averages, you observe the following house effects for three pollsters:

a. Which pollster shows the most credible evidence of a systematic house effect? Calculate a confidence interval for each house effect and explain your reasoning. b. What methodological choices might produce a consistent +4 point Democratic house effect? c. What methodological choices might produce a consistent −2 point Republican house effect? d. If you were building a polling average and had to decide how much weight to give each pollster's polls, what weight would you assign to each and why?

Exercise 10.5 — Trend Analysis

A tracking poll shows the following weekly results over 8 weeks:

a. Calculate the standard error for each weekly margin estimate (MOE for a difference between two proportions). b. A journalist writes: "The Democrat's lead has been collapsing — from +7 in Week 5 to −1 in Week 8, a 8-point swing in just three weeks." Is this a statistically meaningful trend? Explain. c. What is the most reasonable interpretation of the Week 7 and Week 8 results in context of the full series? d. How many polls would you want in a rolling average before trusting a 3-point shift in that average as real movement?

Section B: Python Practice

Exercise 10.6 — Data Loading and Inspection

Load the oda_polls.csv dataset and perform the following analysis:

import pandas as pd
import numpy as np

df = pd.read_csv('oda_polls.csv', parse_dates=['date'])

a. Write code to produce a table showing, for each methodology type: number of polls, average sample size, average margin of error, and average Democratic margin. b. Write code that flags any polls where the reported margin of error is inconsistent with the sample size (i.e., the reported MOE is more than 0.5 percentage points away from the theoretical MOE at 95% confidence for a 50/50 split). c. Create a bar chart showing the distribution of polls by population type (LV, RV, A) and methodology, using a stacked bar chart. d. Write code to identify the five most recent polls and print their key metadata in a clean formatted table.

Exercise 10.7 — Computing Polling Averages

Using the oda_polls.csv dataset:

a. Write a function compute_rolling_average(df, window_days=21, pop_filter='LV') that returns a DataFrame with columns date and avg_margin, containing the rolling average Democratic margin using all polls within window_days of each date, filtered to the specified population.

b. Modify your function to accept a quality_threshold parameter that excludes polls below a certain quality score (using the quality scoring logic from the chapter).

c. Compare the rolling averages using: (1) all LV polls equally weighted, (2) quality-weighted LV polls, and (3) only CATI and Online-Probability LV polls. Plot all three series on the same chart and explain any differences you observe.

d. At what point in the polling timeline would you say the race became consistently "within the margin of error"? Use a confidence interval framework to justify your answer.

Exercise 10.8 — House Effects Analysis

a. Using the calculate_house_effect function from the chapter, extend it to: - Accept a min_concurrent_polls parameter (default 3) - Return confidence intervals for each pollster's house effect alongside the point estimate - Flag pollsters whose house effects are statistically significant at p < 0.05

b. Create a horizontal bar chart of house effects by pollster, with error bars representing 95% confidence intervals. Color bars red for Republican-leaning effects and blue for Democratic-leaning effects.

c. Restrict your house effects analysis to only high-quality polls (CATI and Online-Probability, n ≥ 500). Does the set of statistically significant house effects change? What does this tell you about how methodology differences interact with house effects?

Exercise 10.9 — The Population Definition Comparison

a. Write code to compute, for each week of the polling period, the average Democratic margin among: (1) all adult polls, (2) registered voter polls, (3) likely voter polls. Plot all three as separate lines on the same time-series chart.

b. Calculate the "LV adjustment" — the average gap between registered voter and likely voter polls — and track whether this gap is stable or changing over time. A changing LV adjustment over the campaign period may indicate shifts in turnout enthusiasm.

c. A campaign manager asks: "Should we be looking at the RV or LV polls to track our candidate's actual standing?" Write a 200-word explanation that a non-technical audience would understand.

Section C: Critical Analysis

Exercise 10.10 — Evaluating Real Polls

Find three actual published polls from a current election (using polling databases such as FiveThirtyEight, Ballotpedia, or polling news coverage). For each poll:

a. Identify all AAPOR transparency disclosures that are present or absent. b. Assign a quality score using the framework from Section 10.7.6. c. Identify whether the sponsoring organization has a known political affiliation that might indicate a house effect. d. Compare each poll's result to the concurrent average and estimate the implied house effect. e. Write a one-paragraph assessment of whether each poll's results should be treated as significant evidence about the state of the race.

Exercise 10.11 — The Herding Problem

In the 2020 election cycle, multiple analyses found evidence that pollsters were "herding" — adjusting their results toward consensus. Herding is detectable statistically because the variance across polls is smaller than it should be given their stated margins of error.

a. Explain in your own words why herding is a problem for polling aggregators even if it makes the average appear more accurate in the short term. b. Write pseudocode (or actual Python) that would detect herding in the oda_polls.csv dataset by testing whether the actual variance across contemporaneous LV polls is significantly lower than the expected variance from sampling error alone. c. What incentive structure creates herding behavior, and what institutional change might reduce it?

Section D: Synthesis

Exercise 10.12 — The Analyst's Brief

You are Carlos Mendez. Nadia Osei has asked for a one-page memo summarizing the state of the Garza-Whitfield polling landscape based on your Python analysis. The memo should:

1. State the quality-weighted polling average among likely voters and its associated uncertainty
2. Identify which pollsters have significant house effects and the direction of those effects
3. Explain why the race looks different in adult polls vs. likely voter polls
4. Identify any meaningful trend movement in the last four weeks vs. earlier in the cycle
5. List the three polls Nadia should trust most and explain why

Write this memo as if it will be read by a campaign analytics director who understands polling but does not have time to read the full technical analysis. Maximum 500 words.