Chapter 3: Exercises

Comprehension Check

1. Explain why Daryl Bem's 2011 precognition paper was significant for the replication crisis, even though nobody believed precognition is real.

2. What did the Open Science Collaboration (2015) find when they attempted to replicate 100 published psychology studies? What was the replication rate, and what happened to effect sizes?

3. Define p-hacking and give an example. Why can p-hacking produce statistically significant results even when no real effect exists?

4. What is HARKing? Why does it make the published literature appear more confirmatory than it actually is?

5. Explain the file drawer problem. How does publication bias distort the overall picture of evidence for a given finding?

6. What is the winner's curse in the context of small-sample studies? Why do significant results from small studies tend to overestimate the true effect size?

Application

7. Choose one of the "landmark studies that didn't hold up" from this chapter (ego depletion, social priming, Stanford Prison Experiment, or marshmallow test). Find the original study and one replication attempt on Google Scholar. Compare: - The sample size of the original vs. the replication - The reported effect size of the original vs. the replication - The language used to describe the findings (hedged vs. certain)

8. Go to the Open Science Framework (osf.io) and find three pre-registered studies. For each one, note: - The pre-registered hypotheses - The pre-registered sample size - Whether the results are posted yet How does pre-registration change the incentive structure compared to traditional research?

9. A friend shares a psychology article claiming that "people who eat breakfast are 25% more productive." Apply what you learned in this chapter: what questions would you ask before accepting this claim? (Consider: correlation vs. causation, sample size, publication bias, replication status.)

10. Simmons, Nelson, and Simonsohn (2011) showed that flexible analysis could find "evidence" that listening to a Beatles song makes you younger. Try a thought experiment: if you had a dataset with 100 variables and you ran every pairwise comparison, how many statistically significant results (p < .05) would you expect purely by chance?

Critical Thinking

11. The chapter argues that the replication crisis makes psychology more trustworthy, not less. Do you agree? Make the strongest case for each side.

12. Pre-registration prevents HARKing, but some researchers argue it limits scientific creativity and exploratory discovery. What is the best way to balance confirmatory rigor with exploratory flexibility?

13. If 64% of published psychology findings fail to replicate, what does this mean for the psychology claims in self-help books, corporate training programs, and social media content — most of which are based on pre-crisis research? What should consumers of psychology do with this information?

14. The Stanford Prison Experiment is still taught in most introductory psychology courses. Should it be? If so, how should it be framed? If not, what should replace it?

15. Publication bias means that journals prefer significant results. Registered Reports address this by accepting papers before results are known. What might be the unintended consequences of widespread adoption of Registered Reports?

Fact-Check Portfolio

16. For each of your 10 selected claims, search Google Scholar for the claim plus the word "replication" or "meta-analysis." Record what you find: - Was the original study replicated? - Was the replication successful? - Is there a meta-analysis? - What does the meta-analysis conclude about the effect size? You may find that for some claims, no replication exists — which is itself informative.