Key Takeaways: Hypothesis Testing: Making Decisions with Data

One-Sentence Summary

Hypothesis testing uses indirect reasoning — assume nothing is happening (the null hypothesis), measure how surprising the data would be under that assumption (the p-value), and reject the assumption only if the data are sufficiently surprising (below the significance level $\alpha$) — but the p-value is NOT the probability that the null is true, and "statistically significant" does NOT mean "important."

Core Concepts at a Glance

The Logic of Hypothesis Testing

1. Assume H₀ is true (nothing is happening)
         ↓
2. Collect data and compute test statistic
         ↓
3. Ask: "How surprising are these data under H₀?"
         ↓
4. Compute p-value = P(data this extreme | H₀ true)
         ↓
5. If p-value ≤ α → Reject H₀ (data are too surprising)
   If p-value > α → Fail to reject H₀ (data are plausible under H₀)

The courtroom analogy: - $H_0$ = presumption of innocence - Data = prosecution's evidence - p-value = how convincing the evidence is - $\alpha$ = "beyond a reasonable doubt" threshold - Reject $H_0$ = guilty verdict - Fail to reject $H_0$ = not guilty (NOT the same as innocent)

Test Statistic Formulas

One-Sample z-Test for a Mean (known $\sigma$)

$$\boxed{z = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}}}$$

Conditions: 1. Random sample (or random assignment) 2. Independence (10% condition: $n \leq 0.10 \times N$) 3. Nearly normal population or large sample ($n \geq 30$, by CLT)

One-Sample z-Test for a Proportion

$$\boxed{z = \frac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}}$$

Note: Uses $p_0$ (not $\hat{p}$) in the standard error, because we assume $H_0$ is true.

Conditions: 1. Random sample 2. Independence (10% condition) 3. Success-failure: $np_0 \geq 10$ and $n(1-p_0) \geq 10$

P-Value Calculation

What the P-Value IS and IS NOT

The #1 misconception: "The p-value is the probability the null hypothesis is true." NO. The p-value is $P(\text{data} \mid H_0)$, not $P(H_0 \mid \text{data})$. Confusing these is the prosecutor's fallacy (Chapter 9) applied to statistics.

The Error Matrix

The seesaw: For fixed $n$, decreasing $\alpha$ increases $\beta$ (and vice versa). The only way to reduce both: increase sample size.

One-Tailed vs. Two-Tailed: Decision Guide

Warning: Never switch from two-tailed to one-tailed after seeing the data. That's p-hacking.

The CI-Hypothesis Test Connection

$$\boxed{\text{Reject } H_0: \mu = \mu_0 \text{ at } \alpha = 0.05 \iff \mu_0 \text{ is NOT in the 95\% CI}}$$

The confidence interval contains all values of the parameter that would not be rejected. The hypothesis test tells you whether a specific value is plausible.

The Five Steps (Summary)

P-Hacking and the Replication Crisis

$$P(\text{at least 1 false positive in } k \text{ tests}) = 1 - (1 - \alpha)^k$$

Solution: Pre-register hypotheses. Report all analyses. Use multiple testing corrections (Bonferroni: $\alpha' = \alpha/k$). Distinguish exploratory from confirmatory research.

Python Quick Reference

import numpy as np
from scipy import stats

# --- Manual z-test for a proportion ---
p_hat, p_0, n = 0.38, 0.31, 65
z = (p_hat - p_0) / np.sqrt(p_0 * (1 - p_0) / n)
p_value_right = 1 - stats.norm.cdf(z)         # right-tailed
p_value_left = stats.norm.cdf(z)               # left-tailed
p_value_two = 2 * (1 - stats.norm.cdf(abs(z))) # two-tailed

# --- Manual z-test for a mean (known σ) ---
x_bar, mu_0, sigma, n = 134.2, 130, 20, 64
z = (x_bar - mu_0) / (sigma / np.sqrt(n))
p_value = 1 - stats.norm.cdf(z)  # right-tailed

# --- One-sample t-test (unknown σ, the usual case) ---
data = np.array([...])  # your data
t_stat, p_two = stats.ttest_1samp(data, popmean=mu_0)
# Note: returns TWO-tailed p-value
# For one-tailed: p_one = p_two / 2 (if t in expected direction)

Common Misconceptions

How This Chapter Connects

Threads Resolved

• "P-value explained properly" (from Ch.1): fully delivered
• "What 'statistically significant' means" (from Ch.1): fully delivered
• "Daria's shooting analysis" (from Ch.1): partially resolved (formal test conducted; two-sample framework in Ch.16, power analysis in Ch.17)

The One Thing to Remember

If you forget everything else from this chapter, remember this:

The p-value is the probability of seeing data this extreme or more extreme IF the null hypothesis is true. It is NOT the probability that the null hypothesis is true. A small p-value means the data are surprising under $H_0$ — nothing more, nothing less. "Statistically significant" means $p \leq \alpha$, not "important." These two distinctions — between $P(\text{data} \mid H_0)$ and $P(H_0 \mid \text{data})$, and between statistical significance and practical significance — are the most consequential misunderstandings in all of statistics. Getting them right makes you a better scientist, a better analyst, and a better citizen. Getting them wrong has contributed to a crisis of confidence in published research that we're still recovering from.

Key Terms