Key Takeaways: Power, Effect Sizes, and What "Significant" Really Means

One-Sentence Summary

Statistical significance tells you whether a result is unlikely under the null hypothesis, but it cannot tell you whether the result is important — for that, you need effect sizes, confidence intervals, and power analysis, which together form the complete toolkit for evaluating evidence.

Core Concepts at a Glance

The Key Formulas

Cohen's d (Two Independent Groups)

$$\boxed{d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}, \quad s_p = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}}$$

$r^2$ from a t-test

$$\boxed{r^2 = \frac{t^2}{t^2 + df}}$$

Cohen's d to $r^2$

$$\boxed{r^2 = \frac{d^2}{d^2 + 4}}$$

Cohen's h (Two Proportions)

$$\boxed{h = 2\arcsin(\sqrt{p_1}) - 2\arcsin(\sqrt{p_2})}$$

Statistical Power

$$\boxed{\text{Power} = 1 - \beta = P(\text{reject } H_0 \mid H_0 \text{ is false})}$$

Effect Size Benchmarks

Use with caution: These benchmarks are generic guidelines, not rigid rules. What counts as "small" or "large" depends on the field and the stakes. Always interpret in context.

Four Factors Affecting Power

The Significance Matrix

Key insight: You need both statistical significance and practical significance. A p-value alone cannot tell you which cell you're in.

The Statistical Reporting Checklist

Every statistical analysis should report:

Python Quick Reference

from statsmodels.stats.power import TTestIndPower
from scipy import stats
import numpy as np

# --- Cohen's d ---
def cohens_d(group1, group2):
    n1, n2 = len(group1), len(group2)
    s1, s2 = np.std(group1, ddof=1), np.std(group2, ddof=1)
    s_p = np.sqrt(((n1-1)*s1**2 + (n2-1)*s2**2) / (n1+n2-2))
    return (np.mean(group1) - np.mean(group2)) / s_p

# --- r-squared from t-test ---
def r_squared(t_stat, df):
    return t_stat**2 / (t_stat**2 + df)

# --- Power analysis: find required n ---
power = TTestIndPower()
n_needed = power.solve_power(effect_size=0.5, alpha=0.05,
                              power=0.80, alternative='two-sided')

# --- Power analysis: find achieved power ---
achieved = power.solve_power(effect_size=0.23, nobs1=250,
                              alpha=0.05, alternative='two-sided')

# --- Cohen's h for proportions ---
def cohens_h(p1, p2):
    return 2 * np.arcsin(np.sqrt(p1)) - 2 * np.arcsin(np.sqrt(p2))

Common Misconceptions

How This Chapter Connects

The Key Themes

Theme 4: Uncertainty is not failure. The confidence interval is more honest and more useful than a binary declaration of "significant" or "not significant." Reporting that the true difference is "somewhere between 1 and 8 minutes" communicates both what we know and what we don't. Treating uncertainty as information rather than failure is the hallmark of mature statistical thinking.

Theme 6: P-hacking and ethical data practice. Testing many hypotheses and reporting only the significant ones inflates the false positive rate from 5% to as high as 64%. Publication bias compounds the problem by making the published literature systematically overconfident. The ethical obligation: pre-register hypotheses, report all analyses, report effect sizes, and treat null results as valuable information.

The One Thing to Remember

If you forget everything else from this chapter, remember this:

"Statistically significant" does not mean "important." A p-value tells you whether a result is surprising under $H_0$ — not whether the effect is large, meaningful, or worth acting on. For that, you need the effect size. Cohen's d expresses the difference in standard deviation units (small $\approx$ 0.2, medium $\approx$ 0.5, large $\approx$ 0.8). $r^2$ tells you the proportion of variance explained. Statistical power (minimum 80%) tells you whether you had enough data to find the effect. And the confidence interval — which simultaneously conveys the direction, magnitude, and precision of the effect — is the single most informative summary of any statistical analysis. Always report all of these. Never report just a p-value.

Key Terms