Chapter 20 Exercises: When Models Fail

Conceptual Review

Exercise 20.1 — Systematic vs. Random Error A polling firm surveys 1,200 likely voters and finds the Democratic candidate leading by 4 points. The stated margin of error is ±2.8 points. The actual result is the Republican winning by 1 point.

a) Calculate the polling error in the margin (Democratic minus Republican). b) Is the error within the stated margin of error? What does this tell you? c) If the firm ran 20 polls in different states this cycle, all of which overestimated Democrats by 3–5 points, would you classify this as systematic error, random error, or a combination? Explain your reasoning. d) Why does the stated margin of error not capture systematic bias?

Exercise 20.2 — Correlated State Errors A forecasting model gives a candidate a 90 percent probability of winning based on state polling. The model assumes that state-level polling errors are independent (uncorrelated).

a) Explain in plain language what it means for state errors to be correlated vs. uncorrelated. b) Suppose that in 2016, Pennsylvania, Michigan, and Wisconsin all overestimated Clinton by approximately 5 points. What mechanism could produce identical-direction errors across three states simultaneously? c) If a model assumes independence but errors are correlated, will it overstate or understate the candidate's win probability? Explain why. d) How should a forecaster model correlated state errors? What data would you need to estimate the correlation structure?

Exercise 20.3 — The Herding Problem Read the following hypothetical scenario:

A polling firm's raw data shows Candidate A leading by 6 points. The polling average from other published polls shows Candidate A leading by 2 points. The firm's analyst adjusts the final reported result to +3, reasoning that such a large discrepancy from the consensus is more likely to reflect a methodological error than a genuine signal.

a) What is the name for this behavior? b) Is the firm's reasoning rational from its own perspective? Explain the incentive structure. c) If ten major polling firms all do this, what happens to the informational content of the polling average? d) Describe one structural change to the polling industry that might reduce herding incentives.

Applied Problems

Exercise 20.4 — Postmortem Analysis Using Vivian Park's four-question framework, conduct a brief postmortem of the following scenario:

A forecasting firm predicted a candidate would win with 72 percent probability, with a point estimate of +3.1 percentage points. The actual result was the candidate losing by 1.8 points.

a) State precisely what was predicted (point estimate + probability). b) State precisely what happened. c) Calculate the error in the point estimate. Given a 90-percent confidence interval of approximately ±5 points, is the error within expected statistical variance? d) The firm's postmortem finds that the error was larger in counties with lower poll response rates, which tend to lean Republican. Is the error directional and consistent with a known bias mechanism? What mechanism? e) What methodological change would you recommend?

Exercise 20.5 — International Comparisons Compare the 2016 U.S. presidential polling failure with the 2015 UK general election polling failure.

a) What was the direction of the polling error in each case (which party was overestimated)? b) Identify two structural mechanisms that both failures shared. c) Identify one mechanism that may have been specific to the UK context. d) What does the international pattern of polling failures suggest about explanations that focus on Trump specifically as the cause of American polling failures?

Exercise 20.6 — Partisan Nonresponse Simulation Imagine a true population of 100 voters: 50 Republicans and 50 Democrats. Democrats respond to surveys at a rate of 20 percent; Republicans respond at a rate of 10 percent.

a) How many Democrats and Republicans would you expect in a random sample drawn from this population? b) If the poll is reported without any adjustment, what does it estimate as the Democratic margin? c) What is the true Democratic margin? d) Calculate the polling error. e) If you weight by party registration (you know the true 50/50 split), what margin do you get? What assumption about party registration are you making? f) Under what conditions would party registration weighting fail to fully correct the bias?

Discussion and Critical Thinking

Exercise 20.7 — The Black Swan Question Some analysts argue that 2016 was a "black swan" — an inherently unforeseeable event. Others argue it was a predictable failure given documented methodological problems.

Write a 400-word response that: - Defines what would make an electoral outcome a genuine black swan vs. a predictable failure - Evaluates the 2016 case against this definition using at least two specific pieces of evidence from the chapter - Identifies what you believe is the most important practical implication of the answer for how forecasters should communicate uncertainty

Exercise 20.8 — The Calibration Paradox A forecaster argues: "My models are perfectly calibrated over the past 10 election cycles — when I say 70 percent, I'm right 70 percent of the time. Therefore, my model is doing its job, and I should not change my methodology even though my point estimates were off by 4 points in 2020."

a) Is the forecaster's claim about calibration necessarily inconsistent with having large point estimate errors? Explain. b) What additional information would you need to evaluate whether the 2020 miss was within expected error bounds or reflects a systematic problem? c) Do you agree with the forecaster's conclusion that no methodology change is needed? Why or why not?

Exercise 20.9 — Meridian's Decision At the end of the chapter, Vivian decides to tell clients everything about the methodology error — including that the stated margin of error was based on a nominal sample of 580 rather than the effective sample of 340, making the confidence interval wider than reported.

a) What are the risks of full transparency for Meridian as a business? b) What are the benefits of full transparency, both for Meridian and for its clients? c) What would an ethical framework (from Chapter 38's perspective) say about the obligation to disclose this kind of methodological error? d) Should polling firms be required to report effective sample sizes, or should this be voluntary? Defend your position.

Research Extension

Exercise 20.10 — Comparative Polling Error Analysis Using publicly available data (or data provided in the ODA dataset from Chapter 21), identify all polls conducted in a competitive Senate or gubernatorial race in the most recent election cycle. For each poll:

a) Record the pollster, methodology, sample size, date, and margin estimate. b) Calculate the polling error for each poll (predicted margin minus actual margin). c) Calculate the mean absolute error (MAE) across all polls. d) Test whether the errors have a consistent direction (are systematically positive or negative). e) Look up whether the pollsters used education weighting. Do polls that weight by education show smaller systematic errors in your sample? f) Write a 200-word assessment of what the errors in your sample suggest about the mechanisms discussed in this chapter.