Chapter 10 Quiz: Reading and Evaluating Polls
30 questions — multiple choice, true/false, and short answer
Part I: Multiple Choice (2 points each)
1. The AAPOR Transparency Initiative requires pollsters to disclose which of the following?
a) Predicted election outcome based on their results b) Exact wording of all questions asked c) Individual respondent demographic profiles d) Proprietary likely voter screening formulas
Answer: b. ATI requires exact question wording, among many other disclosures. Proprietary LV screens are not required to be disclosed in full detail; respondent-level data is confidential.
2. A poll shows Candidate A at 51% and Candidate B at 45% with a margin of error of ±4%. A journalist writes "the race is too close to call because A's lead falls within the margin of error." This statement is:
a) Correct, because a 6-point lead with a ±4% MOE means B could actually be winning b) Incorrect, because the 6-point lead is the best available estimate and is more likely to reflect A's actual advantage than a tie c) Correct, because ±4% means A could be anywhere from 47% to 55% d) Incorrect, because the margin of error only applies to the trailing candidate
Answer: b. The "within the MOE" framing treats the confidence interval as a zone of ignorance; the point estimate remains the best single estimate, and a 6-point lead makes A genuinely more likely to be ahead.
3. House effects in polling arise primarily from:
a) Random sampling error that happens to favor one party b) Systematic methodological choices that consistently advantage one party's results c) Interviewer bias introduced by hiring politically active survey staff d) Nonresponse bias specific to campaign polling periods
Answer: b.
4. For two polls of n = 600 each, the margin of error on the difference between polls (i.e., a change in margin) is approximately:
a) The same as the MOE for each individual poll b) Half the MOE for each individual poll c) √2 times the MOE for each individual poll d) 2 times the MOE for each individual poll
Answer: c. The MOE for a difference between two independent proportions equals √2 × MOE_individual.
5. In Python, which pandas method would you use to compute a rolling 14-day average of polling margins?
a) df['margin'].groupby(14).mean()
b) df['margin'].rolling(window=14).mean()
c) df['margin'].resample('14D').mean()
d) df['margin'].expanding(14).mean()
Answer: b.
6. A likely voter screen that only includes respondents who voted in all four of the last four general elections will tend to produce samples that:
a) Are more representative of the full eligible population b) Lean more Democratic because frequent voters are more party-loyal c) Lean more Republican because frequent voters skew older and more conservative in midterm contexts d) Are unbiased because historical behavior is the most reliable predictor
Answer: c. In most midterm contexts, applying strict historical turnout screens skews toward older, more habitual voters who lean Republican.
7. "Herding" in polling refers to:
a) Campaign operatives directing multiple pollsters to field simultaneous surveys b) Pollsters adjusting their results toward the consensus to avoid appearing as an outlier c) Aggregators giving excessive weight to polls that confirm the current average d) Partisan pollsters coordinating their field periods to maximize impact
Answer: b.
8. When comparing a likely voter poll to a registered voter poll of the same race, Republicans typically perform better in the likely voter version because:
a) Republican-leaning voters are more likely to be registered b) The likely voter screen tends to over-represent younger voters who lean Democratic c) Frequent voters, who are overrepresented in LV screens, skew more Republican in midterm contexts d) Registered voter polls are legally required to include non-citizens who lean Democratic
Answer: c.
9. The standard margin of error formula captures which sources of uncertainty?
a) Sampling variability only b) Sampling variability and nonresponse bias c) Sampling variability, nonresponse bias, and weighting uncertainty d) All sources of measurement and sampling error combined
Answer: a. The standard MOE formula captures only sampling variability. It does not capture nonresponse bias, coverage error, measurement error, or weighting uncertainty.
10. In the house effects analysis using Python, why is it important to exclude the target pollster's own polls when calculating the concurrent average?
a) To reduce computational complexity b) To avoid circular reasoning where a pollster's house effect is estimated against an average it contributed to c) Because different pollsters use different date conventions d) To ensure the concurrent average is based only on IVR and CATI polls
Answer: b.
11. Which of the following is the BEST evidence that a published poll has a significant house effect?
a) The poll uses online methodology rather than CATI b) The poll shows results more than 3 points different from a concurrent published poll by another firm c) The pollster consistently shows results in the same partisan direction across multiple independent races and time periods d) The poll has a smaller sample size than competing polls
Answer: c. Consistency across multiple independent races and time periods is the hallmark of a genuine house effect rather than random variation.
12. A subgroup of 80 respondents in a poll of 800 has a margin of error approximately:
a) The same as the full sample MOE b) 10 times larger than the full sample MOE c) √(10) ≈ 3.16 times larger than the full sample MOE d) Half the full sample MOE
Answer: c. MOE ∝ 1/√n, so reducing n by a factor of 10 (from 800 to 80) multiplies the MOE by √10 ≈ 3.16.
13. Vivian Park's 12-question checklist for evaluating polls includes "Who paid for it?" because:
a) Polls funded by governments are subject to different legal standards b) Campaign or advocacy-sponsored polls have incentive structures that may bias results or selective release c) Academic sponsors are required to share data publicly while commercial sponsors are not d) Payer information determines which AAPOR response rate formula applies
Answer: b.
14. Which Python code correctly calculates a quality-weighted polling average using a quality_weight column?
a) df['pct_d'].mean()
b) np.average(df['pct_d'], weights=df['quality_weight'])
c) df.groupby('quality_weight')['pct_d'].mean()
d) (df['pct_d'] * df['quality_weight']).sum()
Answer: b. np.average with a weights parameter computes the weighted mean correctly.
15. The NCPP "20 Questions a Journalist Should Ask About Poll Results" was designed primarily for:
a) Survey researchers designing new questionnaires b) AAPOR certification applications c) Media consumers and journalists interpreting published polls d) Campaign analytics directors evaluating proprietary poll results
Answer: c.
Part II: True/False (1 point each)
16. A poll of all adults is generally the most useful population definition for predicting Senate election outcomes.
Answer: FALSE. Likely voter polls are most predictive of election outcomes. Adult polls include large numbers of non-voters whose preferences may differ substantially from the actual electorate.
17. The margin of error on a 12-point candidate lead in a poll of 600 is the same as the margin of error on a 2-point lead in the same poll.
Answer: FALSE. The MOE formula for the difference between two proportions depends on both proportions: MOE_diff = 1.96 × √(p₁(1-p₁)/n + p₂(1-p₂)/n). For extreme splits (12-point lead = e.g., 56% vs 44%), the MOE is slightly smaller than for a near-50/50 split.
18. In Python, pd.read_csv('oda_polls.csv', parse_dates=['date']) automatically converts the 'date' column to datetime format.
Answer: TRUE.
19. A pollster with a +3 Democratic house effect will always show the Democrat winning even when the Republican is actually ahead.
Answer: FALSE. A house effect shifts estimates systematically, but the direction of the result depends on the true margin. If the Republican is leading by 8 points, a +3 Democratic house effect would show the Republican ahead by 5, not behind.
20. Herding can be detected statistically by testing whether the variance across contemporaneous polls is significantly lower than expected from their stated sample sizes.
Answer: TRUE. If polls are herding, their inter-poll variance will be smaller than the sampling theory prediction, a testable hypothesis.
21. The AAPOR Transparency Initiative is a mandatory certification program that all pollsters must join to conduct public polls.
Answer: FALSE. ATI is voluntary.
22. A rolling average of polls reduces noise relative to individual polls because independent errors partially cancel out.
Answer: TRUE.
23. A poll that shows the same result as the polling average is necessarily more accurate than a poll that deviates from the average.
Answer: FALSE. A poll that deviates from the average might be detecting a real signal that the average has not yet incorporated. Consistency with the average does not confirm accuracy — it may reflect herding.
24. In the house effects framework, a house effect of exactly zero means a pollster is completely unbiased.
Answer: FALSE. A house effect of zero in a specific race or time period means the pollster is consistent with the concurrent average — but the average itself may be biased, and the pollster may have biases on specific question types or subgroups not captured in the topline comparison.
25. matplotlib.dates.DateFormatter('%b %d') formats dates as "Mar 10" style labels in a matplotlib chart.
Answer: TRUE.
Part III: Short Answer (5 points each)
26. Explain in 3–5 sentences why the choice between likely voter and registered voter screens is one of the most consequential methodological decisions in political polling.
Model Answer: Likely voter and registered voter screens measure different populations because not all registered voters turn out to vote, and the gap between the two populations is politically non-random. In most American elections, non-voting registered voters tend to be younger, lower-income, and more Democratic-leaning than frequent voters. Applying a likely voter screen removes these groups preferentially, often shifting results several points in a Republican direction. In a close race, the LV/RV distinction can be the difference between showing one candidate ahead versus behind. The choice of LV model — which indicators to use, how to weight them, what turnout rate to calibrate to — embeds assumptions about turnout that are effectively predictions in themselves.
27. You run the house effects analysis on the ODA dataset and find that "Progressive Polling Inc." has a mean house effect of +3.1 with a standard error of 0.9 across 4 polls. Is this effect statistically significant at the 95% level? Show your calculation.
Model Answer: For a one-sample t-test against zero: t = mean / SE = 3.1 / 0.9 = 3.44. With 3 degrees of freedom (n−1 = 4−1 = 3), the critical t-value at p = 0.05 (two-tailed) is approximately 3.18. Since 3.44 > 3.18, the effect is statistically significant at the 95% level (p < 0.05). There is sufficient evidence to conclude that Progressive Polling Inc. has a genuine systematic Democratic lean in this polling environment.
28. Carlos builds a quality score that assigns Online-Opt-in polls 8 points on the methodology dimension versus 30 for CATI and Online-Probability. A classmate argues this is unfair because "opt-in polls are cheap and fast and often right." How would you respond?
Model Answer: The quality dimension in the scoring system reflects methodological properties that affect statistical validity, not historical accuracy track records. Opt-in polls cannot produce valid statistical margins of error because their sampling is not probabilistic — the population of opt-in panelists is self-selected and not estimably representative of the target population. Even if an opt-in poll happens to be close to the truth in a specific instance, that accuracy cannot be attributed to its design — it could be coincidental. Weighting high-quality and low-quality polls equally because "opt-in polls are sometimes right" is similar to including both scientific measurements and lucky guesses in an average and treating them equally. The quality score penalizes methods that cannot in principle produce valid inference, regardless of whether any particular instance happened to produce an accurate estimate.
29. A campaign's internal poll shows their candidate leading by 8 points. An independent pollster shows the race at 2 points. Both are of registered voters with comparable sample sizes. Without any other information, how should you interpret this discrepancy?
Model Answer: The first step is to ask whether either poll has known house effects — campaign-internal polls consistently show their own candidate performing better because of question ordering, LV model calibration, and (in some cases) selective release of favorable results. The second step is to look at all concurrent polling: is the independent poll consistent with other outside polls? If the independent poll is consistent with others and the internal poll is the outlier, that strongly suggests the internal poll's result reflects methodological choices that favor the campaign. If there are no other concurrent polls, you should weight the independent poll more heavily on methodological grounds (no incentive to bias toward either candidate) while acknowledging the 6-point gap as uncertainty rather than resolving it by splitting the difference. Averaging a biased poll and an unbiased poll produces a biased average.
30. What is the difference between df.groupby('pollster')['margin_d'].mean() and np.average(df['margin_d'], weights=df['quality_weight']), and when would you use each?
Model Answer: df.groupby('pollster')['margin_d'].mean() calculates the simple unweighted mean margin for each pollster separately — useful for analyzing house effects and comparing individual pollsters. It groups first, then averages within groups, returning one value per pollster. np.average(df['margin_d'], weights=df['quality_weight']) calculates a single weighted mean across the entire dataset, where each row's contribution is proportional to its quality weight — useful for computing an overall quality-weighted polling average that gives more influence to better-designed polls. Use the first when you want to compare pollsters to each other (house effects analysis, pollster track records). Use the second when you want a single best estimate of the underlying quantity (the state of the race), where you want to incorporate information about which polls are more reliable.