Case Study 2: Polls, Margins of Error, and the Art of Predicting Elections

The Scenario

It's a Tuesday evening, and Sam Okafor is watching election coverage with his roommates. The anchor announces:

"With polls now closed across the state, we can share the final pre-election poll: Candidate Garcia leads with 52% support, plus or minus 3 percentage points. That means the race is within the margin of error."

"What does 'within the margin of error' even mean?" Sam's roommate Anika asks. "Does it mean the poll is useless?"

Sam, fresh from Chapter 12, knows exactly what it means — and he also knows it's one of the most misunderstood phrases in public discourse.

"It means," Sam says, "that the poll can't tell us who's going to win. But it's telling us a lot more than you think."

Anatomy of a Poll's Margin of Error

What the Numbers Mean

Let's break down the anchor's statement: "52% support, ± 3 percentage points."

Component Value Statistical Name
52% $\hat{p} = 0.52$ Point estimate (sample proportion)
± 3 points $E = 0.03$ Margin of error
(implied) 95% Confidence level Convention in polling
49% to 55% $(0.49, 0.55)$ 95% confidence interval

The 95% confidence level is almost always assumed in U.S. polling — it's the industry standard. When a poll reports "± 3 points," they mean "± 3 points at the 95% confidence level" unless stated otherwise.

Where ± 3 Comes From

Most major national polls use sample sizes of about 1,000 to 1,500 likely voters. Let's see why this produces a margin of error near ± 3 points.

For $n = 1{,}067$ and $\hat{p} = 0.52$:

$$E = z^* \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} = 1.960 \times \sqrt{\frac{0.52 \times 0.48}{1{,}067}} = 1.960 \times \sqrt{0.0002339} = 1.960 \times 0.01529 = 0.02997$$

That's almost exactly 3 percentage points. Not a coincidence — pollsters choose their sample sizes to achieve roughly this margin of error, because ± 3 has become the industry-accepted level of precision.

What "Within the Margin of Error" Means

When the anchor says "the race is within the margin of error," they mean the confidence interval for Candidate Garcia's support includes 50% — the threshold for winning.

  • Garcia's CI: (49%, 55%)
  • Does it include 50%? Yes.

This means we can't be 95% confident that Garcia's true support exceeds 50%. She might be ahead. She might be behind. The data are consistent with both possibilities.

But here's the crucial nuance that the anchor skips: "within the margin of error" doesn't mean the race is a coin flip. Garcia's point estimate is 52%, not 50%. If you had to bet, you'd bet on Garcia. The poll is simply telling you that the lead isn't large enough to be confident it's real.

Garcia's support:

  Definitely      Could go        Garcia
   losing       either way       likely wins

◄─────────┼────────┤═══════╪═══════├──────────►
         45%      49%     52%     55%      60%
                  └───── 95% CI ─────┘
                          │
                         50%
                     (win/lose line)

The History: When Polls Go Wrong

Literary Digest, 1936: The Original Polling Disaster

Sam remembers this from Chapter 4. The Literary Digest magazine mailed questionnaires to 10 million people (!) and got 2.4 million responses. They predicted Alf Landon would defeat Franklin Roosevelt in a landslide. Roosevelt won 46 of 48 states.

The problem wasn't sample size — 2.4 million is enormous. The problem was who responded. The magazine used phone directories and automobile registrations to build their mailing list, systematically excluding people who couldn't afford phones or cars during the Great Depression — people who overwhelmingly supported Roosevelt.

No confidence interval can fix a biased sample. The margin of error formula assumes the sample is random. If it isn't, the CI is centered on the wrong value entirely.

Dewey Defeats Truman, 1948

The Chicago Daily Tribune famously printed the headline "Dewey Defeats Truman" based on early polls and election returns. The polls had stopped polling too early (weeks before the election), missing a late surge for Truman. This led to the development of "likely voter" models and the practice of polling right up to election day.

2016: Clinton vs. Trump

National polls in 2016 predicted Hillary Clinton would win the popular vote by about 3 percentage points. She won it by 2.1 points — well within the margin of error. The national polls were not wrong.

The problem was state-level polls, which systematically underestimated Trump's support in key swing states (Michigan, Wisconsin, Pennsylvania). These polls had margins of error of ± 4 to ± 5 points, and Trump's actual support fell within those margins in most states — but the errors were correlated. When one state poll underestimated Trump, neighboring states tended to as well. This correlation wasn't captured by treating each state as independent.

The lesson: Individual confidence intervals might be valid, but treating them as independent when they're correlated can lead to overconfident predictions. This is a preview of the multiple comparisons problem we'll encounter in later chapters.

Dissecting a Real Poll

Let's work through a realistic polling scenario in detail.

The Setup

A polling firm surveys 1,200 randomly selected likely voters in a state. Results:

  • Candidate Garcia: 624 (52.0%)
  • Candidate Park: 540 (45.0%)
  • Undecided/Other: 36 (3.0%)

CIs for Each Candidate

Garcia: $$\hat{p}_G = 0.520, \quad \text{SE} = \sqrt{\frac{0.520 \times 0.480}{1200}} = 0.01442$$ $$E = 1.960 \times 0.01442 = 0.02827$$ $$\text{95% CI: } (0.492, 0.548) \text{ or } (49.2\%, 54.8\%)$$

Park: $$\hat{p}_P = 0.450, \quad \text{SE} = \sqrt{\frac{0.450 \times 0.550}{1200}} = 0.01436$$ $$E = 1.960 \times 0.01436 = 0.02815$$ $$\text{95% CI: } (0.422, 0.478) \text{ or } (42.2\%, 47.8\%)$$

What These CIs Tell Us (and Don't)

Observation 1: Garcia's CI includes 50%. We can't be 95% confident she'll win.

Observation 2: Park's CI does NOT include 50%. We can be 95% confident that Park's support is below 50%.

Observation 3: The CIs overlap. Garcia's CI goes as low as 49.2%, and Park's goes as high as 47.8%. The overlap region is (47.8%, 49.2%).

Does overlap mean the candidates are tied? This is a common mistake. Overlapping CIs for individual candidates do NOT mean the difference between them is zero. The CI for the difference is a separate calculation that accounts for the shared sample (we'll learn to do this formally in Chapter 16). In this case, the difference is 7 percentage points (52% - 45%), and a proper analysis would show this difference is statistically significant — even though the individual CIs overlap.

💡 Common Mistake Alert

Never compare two groups by checking whether their individual CIs overlap. This is an overly conservative and unreliable test. Instead, construct a CI for the difference between the groups (Chapter 16). Many news analysts make this error on election night.

The Undecided Problem

The 3% undecided voters are the wild card. If they break 2:1 for Park, the race tightens to roughly 52.7% vs. 47.3%. If they break 2:1 for Garcia, it widens to 54.0% vs. 46.0%. Polls can estimate current preferences, but predicting how undecided voters will ultimately vote requires modeling, not just confidence intervals.

How Pollsters Report Margins of Error

When a poll says "± 3 percentage points," there are several things most people don't realize:

1. The MOE Applies to Each Candidate Separately

The stated ± 3 points is the margin of error for any single candidate's percentage. It does NOT mean the lead has a margin of error of ± 3 points. The margin of error for the difference between candidates is actually larger — typically about $\pm 3 \times \sqrt{2} \approx \pm 4.2$ points for the gap between them.

2. The MOE Only Accounts for Random Sampling Error

The margin of error does not include: - Nonresponse bias: People who refuse to answer polls may differ systematically from those who do - Social desirability bias: Respondents may not truthfully report supporting unpopular candidates - Question wording effects: How the question is phrased affects responses - Likely voter model error: Predicting who will actually vote is itself uncertain

These "non-sampling errors" are often larger than the sampling error. The total uncertainty in a poll is almost always greater than the reported margin of error.

3. The MOE Is for the Whole Sample

If the poll reports results for subgroups (e.g., "Among women, Garcia leads 58% to 39%"), the subgroup margin of error is larger than the overall MOE because the subgroup sample is smaller.

For a subgroup of 600 women (half the sample): $$E = 1.960 \times \sqrt{\frac{0.58 \times 0.42}{600}} = 1.960 \times 0.02015 = 0.0395 \approx \pm 4.0 \text{ points}$$

This is larger than the ± 2.8 for the full sample of 1,200. Polls that report subgroup results without adjusting the MOE are misleading.

Simulation: Why "Within the Margin of Error" Isn't "50-50"

Sam runs a simulation to understand what "within the margin of error" really means about the likely outcome:

import numpy as np
from scipy import stats

# Scenario: Garcia at 52% in a poll of 1,200
p_hat = 0.52
n = 1200

# The MOE is about ±2.8 percentage points
z_star = 1.96
se = np.sqrt(p_hat * (1 - p_hat) / n)
moe = z_star * se
print(f"Reported: Garcia {p_hat:.0%}, ± {moe:.1%}")
print(f"95% CI: ({p_hat - moe:.1%}, {p_hat + moe:.1%})")

# Question: If the TRUE support is at various levels within the CI,
# what's the probability the poll would show Garcia at 52%?
# Answer: We can flip this. Given the poll shows 52%, what's
# the probability Garcia ACTUALLY leads (p > 0.50)?

# Under the sampling distribution centered at the observed 52%:
# P(true p > 0.50) ≈ P(Z > (0.50 - 0.52)/SE)
z = (0.50 - p_hat) / se
prob_ahead = 1 - stats.norm.cdf(z)  # P(true p > 0.50)
print(f"\nIf poll shows 52%, probability Garcia truly leads: ~{prob_ahead:.1%}")
# This is about 92% — she's very likely ahead, just not 95% certain!

# Simulation approach: if TRUE p = 0.52 (taking poll at face value),
# how often would she win?
np.random.seed(42)
n_sims = 100_000
wins = 0
for _ in range(n_sims):
    # Simulate the actual election
    votes = np.random.binomial(n=10_000_000, p=0.52)
    if votes > 5_000_000:
        wins += 1
print(f"\nIf true support is exactly 52%:")
print(f"  Garcia wins {wins/n_sims:.1%} of simulated elections")
# She wins essentially 100% of the time if she truly has 52%
# The question is whether 52% is her TRUE support

The punchline: A candidate polling at 52% in a race "within the margin of error" is NOT in a coin-flip race. If her true support really is 52%, she wins essentially every time (in a large electorate, sampling variation is tiny at the election level — millions of voters). The uncertainty is about whether her true support is 52% or something lower.

Statistical models like FiveThirtyEight account for this by converting polling margins into win probabilities. A 52-45 lead with ± 3 MOE typically translates to roughly a 75-85% win probability — far from 50-50, even though it's "within the margin of error."

The Human Side of Margins of Error

There's a reason polls matter beyond academic statistics. They influence:

  • Fundraising: Candidates trailing in polls struggle to raise money, creating a self-fulfilling prophecy
  • Voter turnout: If polls suggest your candidate is far ahead, you might not bother to vote
  • Media coverage: Frontrunners get more coverage, which can itself shift public opinion
  • Policy decisions: Elected officials reference polls to justify their positions

When a poll's margin of error is poorly understood, all of these downstream effects can be distorted. This is why statistical literacy — your statistical literacy, by the end of this course — is a civic skill, not just an academic one.

Python: Polling Analysis

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

# Full polling analysis
n = 1200
garcia = 624
park = 540
undecided = 36

p_garcia = garcia / n
p_park = park / n

# CIs for each candidate
z = stats.norm.ppf(0.975)

for name, count in [("Garcia", garcia), ("Park", park)]:
    p = count / n
    se = np.sqrt(p * (1-p) / n)
    moe = z * se
    print(f"{name}: {p:.1%} ± {moe:.1%}  "
          f"95% CI: ({p-moe:.1%}, {p+moe:.1%})")

# Visualization: Candidate support with CIs
fig, ax = plt.subplots(figsize=(10, 3))

for i, (name, count, color) in enumerate([
    ("Garcia", garcia, 'steelblue'),
    ("Park", park, 'coral')
]):
    p = count / n
    se = np.sqrt(p * (1-p) / n)
    moe = z * se
    ax.errorbar(p, i, xerr=moe, fmt='o', color=color,
                capsize=8, capthick=2, linewidth=2.5,
                markersize=10, label=f'{name}: {p:.1%}')
    ax.text(p + moe + 0.005, i, f'({p-moe:.1%}, {p+moe:.1%})',
            va='center', fontsize=10)

ax.axvline(x=0.50, color='gray', linestyle='--', linewidth=1.5,
           label='50% (win threshold)')
ax.set_yticks([0, 1])
ax.set_yticklabels(['Garcia', 'Park'])
ax.set_xlabel('Support (%)')
ax.set_title('Candidate Support with 95% Confidence Intervals',
             fontweight='bold', fontsize=13)
ax.legend(loc='upper left')
ax.set_xlim(0.35, 0.65)
plt.tight_layout()
plt.show()

# How MOE changes with sample size
print("\n--- How MOE Changes with Sample Size ---")
print(f"{'n':>6}  {'MOE (p̂=0.52)':>14}")
print("-" * 22)
for n_poll in [400, 600, 800, 1000, 1200, 1500, 2000, 5000]:
    se = np.sqrt(0.52 * 0.48 / n_poll)
    moe = z * se
    print(f"{n_poll:>6}  ±{moe*100:>11.1f}%")

Key Takeaways

  1. "Plus or minus 3 points" is a 95% confidence interval for a single candidate's support percentage. It means we're 95% confident the true support falls within that range.

  2. "Within the margin of error" means the CI includes 50% — not that the race is tied. A candidate polling at 52% ± 3 is more likely ahead than behind, just not with 95% certainty.

  3. The reported MOE understates total uncertainty. It only covers random sampling error, not nonresponse bias, social desirability bias, or likely voter model error. Real polling uncertainty is larger.

  4. The MOE for subgroups is larger than for the whole sample. When polls report breakdowns by demographics, each subgroup has less precision.

  5. Polls are snapshots, not predictions. They estimate current opinion within a margin of error. Translating polls into election predictions requires additional modeling.

  6. Sample size determines precision, not representativeness. A random sample of 1,200 is far more reliable than a self-selected sample of 10 million (Literary Digest lesson, Ch.4). The ± 3 MOE only holds if the sample is truly random.

Discussion Questions

  1. A cable news panel debates poll results. Panelist A says "52% to 45% is a real lead." Panelist B says "it's within the margin of error, so it's basically tied." Who is closer to correct, and what nuance are both missing?

  2. Why do most U.S. polls use sample sizes of about 1,000-1,500? What would be the pros and cons of using 10,000?

  3. In 2016, state polls in Michigan showed Clinton ahead by about 3-4 points. Trump won by 0.3 points. Was the poll "wrong"? Was the result "within the margin of error"?

  4. Some countries ban the publication of polls in the final days before an election. Given what you know about how polls affect voter behavior, do you think this is a good policy? What statistical arguments can you make for and against?