Chapter 9 Key Takeaways: Fielding and Data Collection

Chapter 9 Key Takeaways: Fielding and Data Collection

Core Concepts

1. Survey mode is not a neutral choice. Each mode — CATI, IVR, online, mail, face-to-face, SMS — has distinct coverage properties, cost structures, response rate characteristics, and bias sources. The choice of mode shapes who participates and how they respond, making it a consequential analytic decision, not merely an operational one.

2. No single mode provides universal coverage. Landline CATI excludes cell-phone-only households (~25% of U.S. adults). Online opt-in panels exclude non-panelists and the digitally disconnected. IVR legally cannot call cell phones. Multi-mode designs partially address these gaps by combining modes with complementary coverage profiles.

3. AAPOR response rate formulas provide a standardized language for response rate reporting. RR1 through RR6 differ in how they treat partial completions and cases of unknown eligibility. RR1 is the most conservative (lowest); RR6 the most generous. Credible polls report which formula they used. For political polls targeting registered or likely voters, RR3 or RR5 are typically most appropriate.

4. Response rates have declined dramatically — from 70%+ in the 1970s to under 10% today. The primary causes include widespread caller ID adoption, the cell-phone transition, survey fatigue, erosion of civic participation norms, and confusion with telemarketing. This decline is a major methodological challenge for the field.

5. Low response rates do not automatically mean high bias. The critical question is not the response rate itself but whether nonrespondents systematically differ from respondents on the variables of interest. Nonresponse bias = (1 - RR) × (difference between respondents and nonrespondents). A low response rate with random nonresponse produces high variance but not systematic bias.

6. Nonresponse bias from differential participation by political attitude may not be correctable through demographic weighting. The 2020 polling error — systematic overestimation of Biden support — was likely caused partly by Trump supporters' lower propensity to participate in surveys, driven by distrust of professional institutions. Standard demographic weights cannot correct for attitude-based differential response because the attitude is unmeasured.

7. Interviewers change how people answer. Interviewer effects operate through demographic matching (respondents give different answers to interviewers of different race, gender, or political affiliation), expectancy effects (interviewers subtly signal expected answers), and interviewer variance (different interviewers produce systematically different results with the same script). Self-administered modes reduce but do not eliminate these effects.

8. Social desirability bias is real but directionally complex. Respondents report more socially acceptable answers in interviewer-administered modes. The direction of "acceptable" depends on social context — it is not always the liberal or establishment position. SDB is reduced in self-administered online and mail modes.

9. Panel conditioning threatens the validity of longitudinal survey research. Repeated participation makes panelists more politically knowledgeable, more attitudinally crystallized, and more likely to satisfice. Probability panels manage conditioning through survey frequency limits and topic rotation; opt-in panels have less ability to do so.

10. Data cleaning and documentation are analytic decisions with substantive consequences. Choices about which speeders to exclude, how to code ambiguous responses, and how to handle item nonresponse directly shape what the dataset says. These decisions must be pre-specified, documented, and subjected to sensitivity analysis.

Practical Principles

Always identify the mode(s) used in a poll you are evaluating — different modes have different systematic biases.
Ask for the AAPOR response rate when reviewing external polls; if a pollster does not disclose it, treat the results with heightened skepticism.
Check demographic distributions of a completed sample against known population benchmarks (voter files, Census) before accepting topline results.
Require a codebook and field period documentation from any external data provider before building analysis on their dataset.
Report margin of error as sampling error only — explicitly note that it does not capture nonresponse bias, coverage error, or weighting uncertainty.

Connections to Upcoming Chapters

Chapter 10 builds directly on this chapter's discussion of methodology disclosure, applying a checklist for evaluating polls and using Python to detect house effects across the polling landscape.
Chapter 17 (Poll Aggregation) extends the response rate and nonresponse discussion to consider how aggregators should weight polls with different methodology quality.
Chapter 20 (When Models Fail) returns to the 2020 polling error as a case study in the limits of quantitative forecasting.
Chapter 38 (Ethics) examines the ethical obligations of survey researchers to disclose methodology, protect respondent privacy, and resist client pressure to distort results.

The Measurement-Reality Relationship

The organizing theme of this chapter is that measurement shapes the reality it purports to describe. Survey methodology is not a neutral conduit from public opinion to its representation in data. Every operational decision — mode, timing, language, call hours, conversion protocol, cleaning rule — determines whose voice appears in the data and how it is weighted. Good methodology minimizes these distortions; honest disclosure makes them legible to data consumers. Neither goal is fully achievable, but the pursuit of both is what separates professional public opinion research from advocacy dressed in survey clothing.