Further Reading: The Replication Problem

Essential

Ritchie, S. (2020). Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth. Metropolitan Books. The most comprehensive and accessible treatment of the replication crisis across fields. Covers fraud, bias, negligence, and hype in a systematic framework. (Tier 1)

Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False." PLoS Medicine, 2(8), e124. The paper that predicted the replication crisis a decade before it broke. Mathematical, rigorous, and still cited over 12,000 times. (Tier 1)

Open Science Collaboration (2015). "Estimating the Reproducibility of Psychological Science." Science, 349(6251), aac4716. The landmark Reproducibility Project paper. The most systematic replication effort ever conducted. (Tier 1)

P-Hacking and Statistical Problems

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). "False-Positive Psychology." Psychological Science, 22(11), 1359–1366. Demonstrated how standard analytical flexibility can produce p < 0.05 for false claims. Their example: "listening to 'When I'm Sixty-Four' by the Beatles makes you younger." (Tier 1)

Gelman, A. & Loken, E. (2014). "The Statistical Crisis in Science." American Scientist, 102(6), 460–465. Introduced the "garden of forking paths" concept. Accessible and influential. (Tier 1)

Preclinical Research

Begley, C. G. & Ellis, L. M. (2012). "Raise standards for preclinical cancer research." Nature, 483, 531–533. The devastating Amgen replication study. Brief but earth-shaking. (Tier 1)

Goldacre, B. (2012). Bad Pharma. Fourth Estate. Publication bias and replication problems in pharmaceutical research, with a focus on patient harm. (Tier 1)

The Reform Movement

Nosek, B. A. et al. (2015). "Promoting an Open Research Culture." Science, 348(6242), 1422–1425. The manifesto for the Open Science reform movement. Outlines transparency and openness guidelines. (Tier 1)

Research on registered reports suggests they produce substantially different findings than traditional publications — more null results, smaller effect sizes — confirming the magnitude of publication bias in the traditional system. (Tier 2)

The Bem Affair

Bem, D. J. (2011). "Feeling the Future." JPSP, 100(3), 407–425. (Tier 1 — the original precognition paper)

Multiple failed replication attempts are documented in Ritchie, Wiseman, & French (2012) and others. (Tier 2)

For Instructors

The Bem precognition paper makes an exceptional teaching tool. Present the paper's methods and results without revealing the hypothesis. Ask students to evaluate the evidence. Then reveal the claim (precognition). The cognitive dissonance between "the methods look standard" and "the claim is absurd" perfectly illustrates why the replication crisis matters: if standard methods can support precognition, what else are they supporting without warrant?