Chapter 7 Key Takeaways: The Law of Large Numbers — Why Small Samples Lie

The Core Principle

The law of large numbers guarantees that the more times you repeat a random process, the closer your observed average will get to the true underlying probability. This is one of the most powerful and most misunderstood theorems in all of mathematics.

What the Law Says — and Doesn't Say

It says: Observed averages converge to expected values as sample size grows.

It does not say: - Individual outcomes become more predictable - Past outcomes "balance out" future ones (that's the gambler's fallacy) - Your current small sample is reliable - Convergence happens quickly for all types of data

The Two Laws

The weak law of large numbers says that convergence to the true mean becomes arbitrarily probable as sample size grows.

The strong law of large numbers says that convergence to the true mean happens with probability 1 — almost surely.

Both laws reinforce the same practical message: take more samples before trusting a pattern.

Why Small Samples Lie

Small samples are dominated by variance (noise) rather than by the underlying signal. When individual measurements are highly variable — as social media metrics, startup revenues, and test scores tend to be — small samples can show enormous apparent patterns that disappear when more data is collected.

The mathematical formula is clear: the variance of a sample mean equals the population variance divided by sample size (σ²/n). Double your sample size, and you halve the variance of your estimate.

The Social Media Problem

Content creators face a particularly sharp version of the small-sample problem: - Video performance is highly variable (10x or more variation between pieces) - Data arrives fast, creating an illusion of informativeness - Many variables are tested simultaneously (the multiple comparisons problem) - Pre-registration is culturally absent; post-hoc explanation is the norm

Six weeks of content data almost never provides reliable information about what causes performance differences. Treat early findings as hypotheses, not conclusions.

The Multiple Comparisons Problem

If you test 30 variables for correlation with an outcome at the p < 0.05 significance level, you should expect about 1-2 false positives by pure chance — even when nothing is actually correlated. The more variables you test, the more false patterns you will find. This is a fundamental limitation of exploratory data analysis on small samples.

The Hot Hand, Revisited

In genuinely random processes, the hot hand is an illusion. In skill-influenced processes, the hot hand may exist but is much smaller than intuition suggests. Miller and Sanjurjo (2018) showed that earlier studies denying the hot hand contained a mathematical bias — but after correction, the true hot hand effect is modest, not the dominant force that intuition perceives.

The Replication Crisis

The Open Science Collaboration (Nosek et al., 2015) attempted to replicate 100 published psychology studies. Only about 39% replicated as statistically significant. The primary cause: original studies used small samples, producing noisy effect estimates that happened to cross significance thresholds by chance. Replications with more data found smaller, often non-significant, effects.

This is not a scandal of dishonesty. It is a predictable consequence of a field systematically under-sampling.

The Practical Response

The Luck Connection

Many things we attribute to luck, skill, or strategic insight in the short run are simply noise that the law of large numbers hasn't yet averaged out. The "winning strategy" that worked for six weeks, the "talented new hire" who crushed their first month, the "breakthrough approach" that generated three great results — all of these require sustained evidence over enough trials before the signal separates from the noise.

Patience is not passive. It is the active practice of letting the law of large numbers do its work before committing to what you've seen.

Key Terms

Law of large numbers: The theorem stating that observed averages converge to true expected values as sample size increases.

Variance: A measure of how spread out individual measurements are around the mean. Higher variance demands larger samples.

Statistical power: The probability of detecting a real effect when one exists. Low power = high false-negative rate.

Multiple comparisons problem: The increase in false-positive findings when many hypotheses are tested simultaneously.

Pre-registration: Committing to a hypothesis and analysis plan before collecting data — the primary protection against post-hoc false findings.

Replication crisis: The widespread failure of published scientific findings to hold up when independently repeated — largely attributable to small-sample, underpowered original studies.