Chapter 8 Quiz: Sampling: Who Speaks for the Public?

Multiple Choice (10 questions)

1. The Literary Digest's 1936 polling failure is primarily attributed to:

a) Too small a sample — they needed more respondents to accurately reflect the population b) Coverage bias — their sampling frame (auto registrations and phone directories) systematically excluded lower-income voters who favored Roosevelt c) Question wording effects — their survey questions were biased toward Landon d) Nonresponse bias — Republicans were more likely to return the postcard ballots

Answer: b. The Digest used a frame that reached middle-class and wealthy Americans but systematically excluded the working-class and poor voters who were Roosevelt's core supporters. Sample size was enormous; representativeness was catastrophically poor.

2. In stratified sampling, oversampling a subgroup means:

a) Selecting that subgroup at a higher rate than their population proportion b) Giving each member of that subgroup a higher weight in the final analysis c) Excluding that subgroup from the sample to prevent bias d) Both a and b — oversampling requires subsequent downweighting to achieve population estimates

Answer: d. Oversampling a subgroup selects them at a higher rate to ensure sufficient numbers for subgroup analysis, then applies a lower weight when estimating overall population quantities so that the group's contribution to aggregate estimates reflects its true population proportion.

3. The margin of error formula MOE ≈ 1/√n implies that to cut the margin of error in half, you must:

a) Double the sample size b) Triple the sample size c) Quadruple the sample size d) Increase the sample size by 10 times

Answer: c. Because MOE scales with 1/√n, to cut MOE by half you need √n to double — which requires n to increase by a factor of 4. This is the law of diminishing returns in sampling.

4. Which of the following best describes the "sampling frame problem" in political polling?

a) Respondents within a sample may refuse to answer sensitive questions b) The list of population members from which the sample is drawn never perfectly matches the target population, introducing coverage bias c) Sample size calculations are based on assumptions about population variance that may be incorrect d) The margin of error calculation doesn't account for design effects from stratified or cluster samples

Answer: b. The sampling frame problem refers to the gap between the theoretical target population and the practical list from which the sample is drawn. No frame perfectly covers the target population, and the groups that are excluded or underrepresented are often politically non-random.

5. What is the key distinguishing feature of probability sampling that makes it preferable to convenience sampling for political polling?

a) Probability samples always achieve higher response rates b) Probability samples are cheaper and faster to field c) Probability samples give every member of the target population a known, nonzero chance of selection, enabling valid statistical inference d) Probability samples eliminate social desirability bias

Answer: c. The defining feature of probability sampling is known, nonzero selection probabilities for all population members. This property — not size, speed, or cost — is what makes statistical inference from sample to population valid.

6. Post-stratification weighting adjusts the sample to match population benchmarks. Which of the following biases does weighting correct for, and which does it NOT correct for?

a) Corrects for: coverage bias from frame imperfection; Does NOT correct for: random sampling error b) Corrects for: observed demographic imbalances; Does NOT correct for: systematic underrepresentation of groups on unmeasured dimensions c) Corrects for: all nonsampling errors; Does NOT correct for: sampling variance d) Corrects for: question wording effects; Does NOT correct for: order effects

Answer: b. Weighting can correct for imbalances on the variables you can observe and measure (age, race, education, etc.). It cannot correct for systematic differences between respondents and non-respondents on characteristics that are not part of the weighting scheme — the "unmeasured" problem.

7. Multilevel Regression and Poststratification (MRP) is most useful for:

a) Reducing the margin of error in large national samples b) Correcting social desirability bias in sensitive question responses c) Estimating public opinion in small geographies from larger surveys using demographic modeling d) Converting nonprobability samples into probability samples

Answer: c. MRP's primary value is its ability to generate opinion estimates for geographic units (states, congressional districts) that have too few survey respondents for direct estimation. It uses a regression model fit to the larger survey to generate predictions for each demographic cell in each geography, then poststratifies using Census population counts.

8. A telephone poll achieves a 5% response rate. Which statement is most accurate?

a) The results are unreliable because the sample is too small b) The results may be valid if non-respondents are not systematically different from respondents on the variables of interest c) The margin of error should be multiplied by 20 to account for the low response rate d) The poll cannot be published under AAPOR standards

Answer: b. Low response rates do not automatically produce high nonresponse bias. Bias occurs when response propensity is correlated with the outcome of interest. If it is not, a 5% response rate with demographic weighting can produce valid estimates. The key empirical question is whether the responding 5% is systematically different from the non-responding 95% in politically relevant ways.

9. The "design effect" (DEFF) in a cluster sample refers to:

a) The reduction in variance achieved through stratification b) The inflation of variance compared to a simple random sample of the same size, due to within-cluster similarity c) The effect of the question design on response distributions d) The impact of interviewer effects on data quality

Answer: b. The design effect quantifies the statistical cost of cluster sampling. Members of the same cluster are more similar to one another than to the general population, reducing the effective information content of each additional observation within a cluster. A DEFF of 1.5 means your cluster sample is statistically equivalent to an SRS of two-thirds its size.

10. Raking (iterative proportional fitting) is used in survey weighting to:

a) Identify and remove outlier respondents who distort the sample distribution b) Simultaneously match sample marginal distributions to multiple population targets without requiring knowledge of the full joint distribution c) Convert cluster samples into stratified samples for analysis d) Eliminate coverage bias by adjusting the sampling frame before data collection

Answer: b. Raking iteratively adjusts weights across multiple weighting variables (age, gender, race, etc.) until all marginal distributions match their population targets. Its advantage over simple post-stratification is that it doesn't require knowing the full joint distribution of all weighting variables — only the margins.

Short Answer (5 questions)

11. Explain the difference between coverage bias and nonresponse bias. Give a specific example of each in the context of political polling.

Model answer: Coverage bias occurs when certain members of the target population have zero probability of being selected — they are not in the sampling frame at all. For example, a telephone survey that uses only landline random-digit-dialing excludes the approximately 60% of American households that are cell-phone-only. These households skew younger and more diverse; their systematic exclusion introduces coverage bias in the estimate of population opinion. Nonresponse bias occurs when members of the population are in the sampling frame but decline to participate — their selection probability is nonzero, but their response probability is zero (or very low). For example, if politically disengaged voters are much less likely to respond to a political survey than politically engaged voters, the resulting sample will overrepresent the engaged and produce an unrepresentative picture of overall public opinion. Coverage bias is fixed at the frame stage; nonresponse bias occurs at the contact/cooperation stage.

12. What is a "likely voter screen," and why does the choice of screening criteria matter for poll results?

Model answer: A likely voter screen is a set of criteria used to identify, from a larger sample of registered or eligible voters, those who are most likely to actually vote in the upcoming election. The two main approaches are intent-based screens (asking respondents how likely they are to vote) and history-based screens (using voter file records of past participation). The choice matters enormously because likely voters systematically differ from the broader electorate. They are, on average, older, whiter, more educated, and more partisan than either registered voters or all adults. If one candidate's coalition relies more heavily on lower-propensity voters — younger voters, sporadic voters, or newly registered voters — a strict likely voter screen will understate that candidate's true potential support. Different screening criteria can produce horse-race estimates that differ by several percentage points for the same race at the same time, even from the same underlying sample.

13. Explain why the Literary Digest example demonstrates that "bigger is not better" in sampling. What is better than bigger?

Model answer: The Literary Digest mailed 10 million postcards and received 2.4 million responses — numbers that dwarf any contemporary political poll. Yet the result was catastrophically wrong, while George Gallup correctly predicted the winner with a sample of roughly 50,000. The lesson is that precision and validity in survey research come from representativeness, not from sample size. A large, biased sample will be biased with high precision — it will consistently produce the wrong answer. A smaller, well-drawn probability sample will be accurate with appropriate uncertainty. "Better than bigger" means: a sample drawn from a frame that covers the target population, using probability sampling that gives all members a known chance of selection, and weighted to correct for any demographic imbalances. Thousands of well-selected respondents outperform millions of self-selected ones.

14. What is the intuition behind MRP, and when is it most useful?

Model answer: MRP — Multilevel Regression and Poststratification — works in two steps. First, you use your survey data to build a statistical model that predicts the opinion of interest (e.g., candidate support) as a function of individual demographic characteristics (age, race, education, gender) and geographic characteristics (region, state). Second, you use that model to generate predicted opinions for every combination of demographic characteristics in every geographic unit, then weight those predictions by the actual Census counts for each cell. The result is an estimate of average opinion in each geographic unit that combines your direct survey data with systematic demographic patterns. MRP is most useful when you want opinion estimates for many small geographies — congressional districts, counties, state legislative districts — that individually have too few survey respondents for direct estimation. It is not magic: it produces better estimates than direct small-n estimates, but it depends on the demographic patterns captured in the model being the dominant source of geographic variation in opinion.

15. What is the "spiral of silence" and how does it relate to the declining response rate crisis in modern polling?

Model answer: The spiral of silence, as developed by Noelle-Neumann, describes the tendency of people who perceive their opinions to be in the minority to withhold public expression of those opinions. In the polling context, if respondents who hold politically stigmatized views are less willing to express them — or less willing to participate in surveys at all — the low response rates of contemporary polling may amplify this bias. Some researchers have attributed polling errors in elections involving controversial candidates partly to a "shy voter" effect: supporters of candidates associated with socially stigmatized positions may be underrepresented among the fraction of contacted people who agree to participate. The declining response rate crisis exacerbates this problem because the smaller the responding fraction, the more consequential it is if that fraction is systematically different from non-respondents on politically relevant characteristics. At a 60% response rate, a small spiral-of-silence effect might be negligible; at a 5% response rate, the same effect could introduce substantial bias in the direction of underestimating support for the stigmatized candidate.