Key Takeaways: The Replication Problem

The Big Idea

The foundational evidence behind many scientific consensuses has never been independently verified — because the incentive structures of every field systematically discourage the boring, unglamorous, career-damaging work of checking. The system is optimized for production of findings, not verification of findings.

Core Concepts

The Three Practices That Inflate False Positives

1. P-hacking — Analyzing data multiple ways until p < 0.05 appears
2. HARKing — Hypothesizing After Results are Known
3. Researcher degrees of freedom — The "garden of forking paths" of analytical choices

The Four Structural Incentives Against Replication

1. Publish-or-perish (replications don't count for careers)
2. Journal novelty bias (replications rejected as "not novel")
3. Social cost (challenging colleagues is risky)
4. Funding (grants don't support replication)

Replication Rates Across Fields

The Evidence Hierarchy

Replicated findings > Pre-registered findings > Large-sample findings > Single published studies

Ioannidis's Key Insight

Given publication bias, small samples, researcher degrees of freedom, and low prior probability, the majority of published "significant" findings may be false positives — not because of fraud but because of the mathematics of the system.

Epistemic Audit — Chapter 10 Addition

After this chapter: assess replication status of core claims, evaluate replication culture, identify researcher degrees of freedom, estimate what it would cost to replicate the key findings.

What's Coming Next

Chapter 11: How Incentive Structures Manufacture Error — the business model of being wrong.

Quick Reference:

"Has this been replicated?"
   YES → Higher confidence
   NO  → "Why not?"
         → Because nobody tried (structural disincentive)
         → Because replications failed (finding may be false)
         → Because replication is impossible (different concern)