Exercises: The Replication Problem

Difficulty Guide: - ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Challenging | ⭐⭐⭐⭐ Advanced/Research

Part A: Conceptual Understanding ⭐

A.1. Define p-hacking, HARKing, and researcher degrees of freedom. For each, explain why it is a structural incentive problem rather than individual misconduct.

A.2. The Reproducibility Project found that 36% of psychology studies replicated. Does this mean that 64% of psychology is "wrong"? Why or why not?

A.3. Explain Ioannidis's argument for why most published research findings are false. What are the key variables in his model?

A.4. Why is replication structurally disincentivized? List the four incentives and explain how each one pushes against verification.

A.5. What is a registered report? How does it address publication bias?

A.6. Distinguish between "the claim is false" and "the specific study supporting the claim is unreliable." Why does this distinction matter for interpreting the replication crisis?

Part B: Applied Analysis ⭐⭐

B.1. The worked example in section 10.2 demonstrates how p-hacking produces a false "Mozart effect" finding. Design your own worked example using a different research question. Show how a null result can be transformed into a "significant" finding through researcher degrees of freedom.

B.2. Choose a famous finding in your field. Assess its replication status: Has it been independently replicated? By how many groups? With what effect sizes? If it hasn't been replicated, why not?

B.3. Apply Ioannidis's framework to your field. Estimate (roughly) the prior probability of hypotheses, typical sample sizes, typical power, and likely degree of publication bias. What does the model predict about the reliability of your field's published findings?

B.4. Compare the replication problem in psychology (36% replication) with preclinical cancer research (11% replication). Why might the rates differ? What structural features explain the difference?

B.5. The Bem precognition paper was published in a top journal using standard methods. What does this tell us about the methods themselves?

Part C: Research Design Challenges ⭐⭐–⭐⭐⭐

C.1. Design a career evaluation system for academic researchers that incentivizes replication and open science without destroying the incentive for original research.

C.2. Propose a funding mechanism that would allocate resources to systematic replication in your field. How much would it cost? How would you prioritize which findings to replicate?

C.3. Design a "replication audit" for your field: a systematic assessment of how much of the foundational evidence has been independently verified.

Part D: Synthesis & Critical Thinking ⭐⭐⭐

D.1. The replication crisis is sometimes described as "science working as it should" (errors being caught and corrected) and sometimes as "science failing" (errors not being caught for decades). Which framing is more accurate? Argue both sides.

D.2. How does the replication problem interact with all the previous failure modes? Trace the interaction of replication failure with authority cascade (Ch.2), unfalsifiability (Ch.3), streetlight effect (Ch.4), survivorship bias (Ch.5), plausible story problem (Ch.6), anchoring (Ch.7), imported error (Ch.8), and sunk cost (Ch.9).

D.3. If Ioannidis is right that most published findings are false, what should a policymaker do? Ignore all published research? Fund massive replication programs? Something else?

Part M: Mixed Practice (Interleaved) ⭐⭐–⭐⭐⭐

M.1. (From Chapter 4) How does Goodhart's Law apply to the p < 0.05 threshold? What happened when "significance" became a target?

M.2. (From Chapter 5) Publication bias selects positive results. P-hacking inflates false positive rates. How do these two mechanisms interact to distort the evidence base?

M.3. (From Chapter 9) The sunk cost of the methodological infrastructure makes the replication crisis harder to address. Map the five switching cost components for p-value-based significance testing.

M.4. (Integration) You now have ten diagnostic lenses. Update your Epistemic Audit with the replication assessment.

Part E: Research & Extension ⭐⭐⭐⭐

E.1. Read Ioannidis's "Why Most Published Research Findings Are False" (2005). Summarize the argument and evaluate its assumptions.

E.2. Read Ritchie's Science Fictions (2020). Write a 1,500-word review focusing on what the book adds to this chapter's analysis.

Solutions

Selected solutions in appendices/answers-to-selected.md.