Chapter 20 Quiz: When Models Fail
Multiple Choice
1. In the 2016 presidential election, the polling error in the national popular vote was approximately:
a) 0.5 percentage points (Clinton overestimated) b) 1.0 percentage points (Clinton overestimated) c) 3.9 percentage points (Clinton overestimated) d) 5.0 percentage points (Trump underestimated in each state)
2. "Correlated state errors" refers to a situation where:
a) Polling errors in different states are statistically independent b) Polling errors in different states run in the same direction for the same underlying reasons c) The margin of error is larger in some states than in others d) State-level polls are conducted by different firms with incompatible methodologies
3. Which of the following is NOT one of Vivian Park's four postmortem questions?
a) What did we predict, precisely? b) What happened, precisely? c) Did our competitor's model also miss? d) Was the error directional and consistent with a known bias mechanism?
4. The "herding" problem in polling occurs when:
a) Multiple candidates split the vote, creating horse-race dynamics b) Polling firms adjust their results toward industry consensus to avoid being outliers c) Voters give false answers to polling questions to mislead campaigns d) A single well-resourced pollster dominates the public polling market
5. Partisan nonresponse bias is difficult to correct with education weighting because:
a) Education data is not available to polling firms b) The bias operates at the party level independently of demographic characteristics c) Education weighting always makes the sample less representative d) Courts have prohibited education-based weighting in political surveys
6. In the 2015 UK general election, the polling error:
a) Overestimated the Conservatives b) Accurately predicted a hung parliament c) Overestimated Labour d) Underestimated the Liberal Democrats
7. Systematic error in polling differs from random error in that it:
a) Produces larger margins of error on published polls b) Is corrected by increasing sample size c) Runs consistently in one direction rather than averaging to zero d) Is most common in online panels rather than telephone surveys
8. The 2022 midterms were notable for producing:
a) The largest polling error in history, favoring Republicans b) A polling miss that ran in the Democratic direction, against the expected "red wave" c) Highly accurate polling that validated post-2016 methodological changes d) A split result in which Senate polls were accurate but House polls were not
9. Which of the following most accurately describes what the stated margin of error in a published poll captures?
a) The full range of possible polling error, including systematic biases b) Only the sampling variance from drawing a finite sample using a correct methodology c) The maximum possible error due to both sampling variance and weighting d) The 95-percent confidence interval accounting for nonresponse
10. The Australian 2019 polling failure is relevant to understanding American polling failures because it demonstrates that:
a) International observers can predict American elections more accurately than domestic pollsters b) Polling failures appear across different countries, suggesting structural industry-wide problems c) Presidential systems are more predictable than parliamentary systems d) Online polling is always less accurate than telephone polling
True/False
11. The 2020 polling error was smaller than the 2016 polling error in terms of raw magnitude.
12. Weighting on recalled vote choice perfectly corrects for partisan nonresponse bias.
13. A model can be well-calibrated (accurate about stated probabilities) while still having significant directional bias in its point estimates.
14. Brexit is typically described as a polling failure because polls showed a clear Remain advantage that did not materialize.
15. Effective sample size is always less than or equal to nominal sample size when survey weights are applied.
Short Answer
16. In two or three sentences, explain why a model that assumes independent state-level polling errors will tend to give overly confident win probabilities when the true errors are correlated.
17. Vivian Park distinguishes between explaining "why the candidate lost" and explaining "why our model of the electorate was wrong." In one to two sentences, explain why this distinction matters for the purpose of a postmortem.
18. Define "differential nonresponse" and explain in one to two sentences how it can produce systematic polling error even when a firm uses sophisticated probability sampling.
19. What is the "winner's memory effect" and why does it complicate the strategy of weighting polling samples on recalled vote choice?
20. The chapter argues that "some events are genuinely difficult to forecast not because of methodological failures but because they involve rare combinations of circumstances." Give one specific example from the chapter and explain in two sentences why it supports this claim rather than the methodological failure explanation.
Answer Key
- b
- b
- c
- b
- b
- c
- c
- b
- b
- b
- False (2020 national error was approximately 3.9 points vs. 1.0 point in 2016)
- False (it introduces its own problems — vote recall bias and assumes stable party composition)
- True
- False (final polls were essentially tied; the failure was interpretive overconfidence, not polling inaccuracy)
- True