Chapter 17 Quiz: Poll Aggregation

Multiple Choice

1. The primary statistical reason poll aggregation improves on individual polls is that it:

a) Eliminates all sources of polling error b) Reduces random error by averaging independent measurements c) Corrects for systematic biases through mathematical adjustment d) Gives the most accurate poll higher weight

2. A "house effect" in polling refers to:

a) The overrepresentation of homeowners in polling samples b) The systematic partisan lean of a specific pollster across multiple polls c) The effect of domestic issues on poll results d) The tendency of online polls to favor one party over phone polls

3. RealClearPolitics and FiveThirtyEight differ primarily in that:

a) RCP includes only Republican-leaning polls; 538 includes only Democratic-leaning polls b) RCP uses a simple average of recent polls; 538 uses weighted averages with quality and recency adjustments c) 538 is government-funded; RCP is private d) RCP adjusts for house effects; 538 does not

4. "Herding" in polling refers to:

a) The practice of commissioning many polls simultaneously b) Pollsters deliberately adjusting their results toward the polling average to avoid being an outlier c) The tendency of voters to follow the crowd when forming opinions d) Aggregators excluding outlier polls from their averages

5. Recency weighting in poll aggregation is based on the principle that:

a) Recent polls are always more accurate than older polls b) Older polls should be excluded entirely from averages c) Opinion can shift over time, making more recent polls more informative about current opinion d) Polling error decreases as Election Day approaches

6. Which of the following poll characteristics would NOT typically affect quality weighting in an aggregation model?

a) Historical pollster accuracy vs. actual election results b) Whether the poll discloses full methodology and crosstabs c) Whether the poll uses probability-based or opt-in sampling d) The political party affiliation of the pollster's founder

7. The aggregation ecosystem "influence problem" refers most precisely to:

a) The difficulty of convincing campaigns to use aggregated data b) The potential for aggregator ratings to shape campaign resource allocation and voter behavior c) The influence of wealthy donors on which polls are published d) The way media coverage influences poll results

8. A pollster uses a strict likely voter screen that historically produces results 2.5 points more Republican than the actual election outcome. An aggregator that adjusts for house effects would:

a) Exclude this pollster's results entirely b) Double the sample size weight for this pollster's results c) Shift this pollster's results 2.5 points in the Democratic direction before averaging d) Weight this pollster's results at zero

9. If ten truly independent polls all have a margin of error of ±4 points, the margin of error on their combined average would be approximately:

a) ±4 points (unchanged) b) ±1.3 points c) ±0.4 points d) ±40 points

10. Which of the following is the most accurate statement about the limits of poll aggregation?

a) Aggregation becomes unnecessary once you have more than five polls b) Aggregation can reduce random error but cannot correct systematic errors that affect all polls simultaneously c) Aggregation is only useful for presidential races, not down-ballot races d) Once a polling average is published, it becomes self-fulfilling

True/False

11. A polling average that includes twenty polls will always be more accurate than an average that includes five polls. (True/False — explain briefly)

12. Cook Political Report and Sabato's Crystal Ball use roughly the same methodology as FiveThirtyEight. (True/False)

13. Herding, if widespread, would reduce the apparent diversity of polling results below what we'd expect from random sampling variation. (True/False — explain)

14. Aggregators can fully correct for a situation where every pollster in a state underestimates one candidate's support. (True/False — explain)

15. A Democratic-leaning house effect always means a pollster is being deliberately biased. (True/False — explain)

Short Answer

16. Explain in two to three sentences why poll aggregation reduces random error but does not reduce systematic error.

17. What is the difference between a "polls-only" model and a "polls-plus" model in the FiveThirtyEight methodology?

18. A campaign manager dismisses all aggregators, saying "our internal poll is the most accurate because it uses our proprietary voter file." What statistical argument would you make for still consulting the aggregators?

19. Describe one way that publishing a polling average could change the outcome of the election being measured.

20. What is the "Economist's model" contribution to the aggregation ecosystem, and how does it differ from FiveThirtyEight's approach to transparency?

Answer Key

  1. b
  2. b
  3. b
  4. b
  5. c
  6. d
  7. b
  8. c
  9. b
  10. b
  11. False — if the additional polls have lower quality or higher house effects, adding them can decrease accuracy; more polls from the same systematically biased source worsen precision without improving accuracy.
  12. False — Cook and Sabato use qualitative-editorial judgments integrated with polling data; FiveThirtyEight uses quantitative weighting algorithms.
  13. True — if pollsters are herding toward a consensus estimate, their results will cluster more tightly than statistical sampling variation alone would produce.
  14. False — aggregation reduces random variation across polls but cannot correct for systematic error that affects all polls in the same direction; averaging many biased polls gives a very precise biased estimate.
  15. False — house effects often arise from legitimate methodological choices (LV screen design, weighting procedures, question ordering) and do not imply intentional bias.
  16. See section 17.1.
  17. See section 17.3.
  18. One poll of 600-800 respondents has a MOE of ±3-4 points; a ten-poll average has an effective MOE of ±1-2 points. The internal poll could be showing a favorable tail of the distribution; the aggregate is more likely near the true center.
  19. Candidates rated as heavy favorites may see supporters stay home (turnout suppression); campaigns may redirect resources away from "safe" races; media coverage may entrench ratings as conventional wisdom.
  20. The Economist's model publishes full code and methodology, offering a transparency-complexity compromise; FiveThirtyEight's full weighting formulas are proprietary.