Key Takeaways: Comparing Two Groups

One-Sentence Summary

Comparing two groups — the most common analysis in applied statistics — requires choosing between the two-sample t-test (independent means), the paired t-test (dependent means), or the two-proportion z-test (independent proportions), each following the same fundamental logic: measure the difference, compute the standard error of that difference, and determine whether the difference is too large to be explained by chance.

Core Concepts at a Glance

The Three Tests

1. Two-Sample t-Test (Welch's)

$$\boxed{t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}}$$

• When: Comparing means from two independent groups
• df: Welch-Satterthwaite approximation (let software compute it)
• Default choice: Welch's (does not assume equal variances)
• Example: Alex's A/B test — old algorithm vs. new algorithm

2. Paired t-Test

$$\boxed{t = \frac{\bar{d}}{s_d / \sqrt{n}}, \quad df = n - 1}$$

• When: Data come in natural pairs (before/after, matched subjects)
• Key insight: This IS a one-sample t-test on the within-pair differences
• Why it's powerful: Eliminates between-subject variability
• Example: Sam's game-by-game comparison of Daria's scoring

3. Two-Proportion z-Test

$$\boxed{z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}_{\text{pooled}}(1 - \hat{p}_{\text{pooled}})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}}$$

• When: Comparing proportions from two independent groups
• Pooled proportion: $\hat{p}_{\text{pooled}} = (X_1 + X_2)/(n_1 + n_2)$
• Note: Test uses pooled SE; CI uses unpooled SE
• Example: James's recidivism rate comparison

The Decision Flowchart

     Comparing two groups?
            │
      What type of variable?
      ╱                 ╲
  Numerical          Categorical
   (means)          (proportions)
      │                  │
   Paired?         Two-proportion
   ╱     ╲            z-test
  Yes     No
   │       │
Paired  Two-sample
t-test   t-test
         (Welch's)

Confidence Intervals for Differences

Reading the CI: - Contains zero → no significant difference - Entirely positive → Group 1 plausibly higher - Entirely negative → Group 1 plausibly lower - Width → precision of the estimate

Conditions Summary

Python Quick Reference

from scipy import stats
from statsmodels.stats.proportion import proportions_ztest
import numpy as np

# --- Two-sample t-test (Welch's, default) ---
t, p = stats.ttest_ind(group1, group2, equal_var=False)
# One-tailed:
t, p = stats.ttest_ind(group1, group2, equal_var=False,
                        alternative='greater')

# --- Paired t-test ---
t, p = stats.ttest_rel(after, before)
# Equivalent:
diffs = after - before
t, p = stats.ttest_1samp(diffs, popmean=0)

# --- Two-proportion z-test ---
count = np.array([successes1, successes2])
nobs = np.array([n1, n2])
z, p = proportions_ztest(count, nobs)

Excel Quick Reference

Common Misconceptions

How This Chapter Connects

The Key Themes

Theme 2: Comparing groups is where bias becomes visible. Every fairness audit, pay equity analysis, and health disparity study uses two-group comparisons. James's false positive rate comparison revealed algorithmic bias that the overall comparison masked. The methods in this chapter are tools for justice as much as tools for science.

Theme 5: Correlation vs. causation in group comparisons. Finding a significant difference between two groups does not automatically mean one group's condition caused the difference. Alex's randomized A/B test supports a causal claim. Maya's observational comparison reveals an association that demands further investigation. The statistical test tells you the difference is real. The study design determines what kind of "real" it is.

The One Thing to Remember

If you forget everything else from this chapter, remember this:

When comparing two groups, ask two questions before choosing a test: (1) Am I comparing means or proportions? (2) Are the data independent or paired? The answers determine whether you use a two-sample t-test, a paired t-test, or a two-proportion z-test. All three follow the same logic: observed difference divided by standard error of the difference. Use Welch's t-test by default for independent means (it doesn't assume equal variances). For paired data, compute within-pair differences and run a one-sample t-test. Always report the confidence interval for the difference alongside the p-value — the CI tells you HOW BIG the difference is, not just WHETHER it exists. And remember: the test tells you the difference is real; the study design tells you whether you can call it causal.

Key Terms