Further Reading: Adversarial Collaboration and Other Tools

Tier 1: Verified Sources

Chambers, Chris. The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice. Princeton University Press, 2017. The definitive case for registered reports and other Open Science reforms, by the psychologist who has done the most to develop and promote the registered report format. Chambers provides both the diagnosis (publication bias, p-hacking, HARKing) and the prescription (pre-registration, registered reports, open data).

Kahneman, Daniel. "A Proposal to Deal with Questions about Priming Effects." Open letter, 2012. Kahneman's letter to the social priming research community proposing adversarial collaboration as a mechanism for resolving disputes about replication. A landmark document in the history of the replication crisis.

Nosek, Brian A., et al. "Promoting an Open Research Culture." Science 348, no. 6242 (2015): 1422-1425. The Transparency and Openness Promotion (TOP) Guidelines — a framework of eight standards for transparent research, each with three levels of implementation. Provides a concrete roadmap for implementing several of the tools in this chapter.

Dreber, Anna, et al. "Using Prediction Markets to Estimate the Reproducibility of Scientific Research." Proceedings of the National Academy of Sciences 112, no. 50 (2015): 15343-15347. The first systematic test of whether prediction markets can accurately predict replication outcomes. Found that markets were reasonably accurate — supporting the concept's viability while acknowledging limitations of scale.

Zenko, Micah. Red Team: How to Succeed by Thinking Like the Enemy. Basic Books, 2015. A comprehensive account of red teaming across military, intelligence, and corporate contexts. Zenko documents both successes and failures, providing the evidence base for the chapter's assessment that red team effectiveness depends entirely on institutional culture.

Tier 2: Attributed Claims

The finding that registered reports produce approximately 55-60% null results (compared to ~5-10% in traditional publications) has been documented in analyses by Chris Chambers and others, based on the growing corpus of published registered reports across multiple fields.

The Center for Open Science (cos.io), founded by Brian Nosek in 2013, provides infrastructure for pre-registration (the Open Science Framework) and tracks adoption of Open Science practices across journals and fields.

The Dutch science funding agency NWO's Replication Studies program is documented in NWO's public reports and in published analyses of the program's outcomes.

The U.S. Army's University of Foreign Military and Cultural Studies (UFMCS) at Fort Leavenworth, Kansas, established in 2004, is the primary institutional home of military red team training. Its curriculum and approach are publicly documented.

  1. Start with Chambers (Seven Deadly Sins) — for the comprehensive case for reform
  2. Then Nosek et al. (2015) — for the concrete implementation framework
  3. Then Dreber et al. (2015) — for the prediction markets evidence
  4. Then Zenko (Red Team) — for the red team analysis
  5. Then Kahneman (2012 letter) — for the adversarial collaboration proposal in its original context