Chapter 40 Exercises: Critical Thinking About Attraction Research

Exercise 1: Running and Interpreting the P-Hacking Simulation

Open code/meta_analysis_tools.py and run it in your Python environment. After examining the output:

a) The simulation uses 10 analytic choices per researcher. Modify the n_analyses parameter to 1, 3, 5, and 20. Record the false positive rate at each value. Plot these results (or describe the pattern) and explain what they reveal about the relationship between analytic flexibility and false discovery rate.

b) Now modify n_participants from 100 to 50, then to 500, keeping n_analyses at 10. How does sample size interact with p-hacking? Why does this relationship make sense theoretically?

c) Based on your experiments: a researcher tells you that their study used a "reasonable" exclusion criterion (removing participants who finished in the bottom 5% of response times) and found p = .047. They did not pre-register. How should you interpret this finding? What additional information would change your assessment?

Exercise 2: Forest Plot Interpretation

Using the forest plot generated by the simulation (or a forest plot from a real published meta-analysis assigned by your instructor):

a) Identify three studies whose confidence intervals cross the null (d = 0) line. What does this mean for those individual studies?

b) Look at the diamond at the bottom. Is the meta-analytic estimate positive, negative, or near zero? Does the diamond cross the null line? What does each case imply?

c) The chapter states that I² = 0% means all variation is sampling error, while I² above 75% suggests substantial heterogeneity. Your simulated forest plot shows a specific I² value. What does this suggest about the meaning of the pooled estimate? Should you trust the pooled effect as a universal fact, or as an average across genuinely different effects?

d) Suppose you learn that six of the fifteen studies in this meta-analysis were conducted by the same research group. How might this affect your interpretation of the pooled effect?

Exercise 3: Funnel Plot Analysis

Compare the two funnel plots generated by meta_analysis_tools.py (symmetrical vs. biased).

a) Describe in your own words what distinguishes the two plots visually.

b) Why does a missing lower-left quadrant (small studies with small or negative effects) specifically indicate publication bias rather than just normal variation?

c) Egger's test formally tests for funnel asymmetry. Look up Egger's test (the 1997 paper by Egger et al. in the BMJ is freely available). Write a 200-word explanation of how it works and what a statistically significant result means.

Exercise 4: The Press Release Audit

Find a press release about an attraction or relationship science finding from the last two years (university press offices, EurekAlert, and ScienceDaily are good sources). Apply the seven-step press release evaluation protocol from Section 40.9:

  1. Find the original study
  2. Check the sample
  3. Find the effect size
  4. Check the preregistration status
  5. Look for the limitations section
  6. Look for replication evidence
  7. Check the causal language

Write a 600–800 word audit. Quote specific language from both the press release and the original study to support your analysis. The goal is not to dismiss the finding but to accurately characterize the confidence it warrants.

Exercise 5: Designing a Replication

Choose any finding from Chapters 6–38 of this book that you believe deserves replication in a more diverse sample. Write a brief pre-registration document (400–500 words) that includes:

  • The specific hypothesis you would test (stated precisely enough that another researcher could judge whether your final result confirms it)
  • The population you would sample and why it addresses the original study's sampling limitation
  • The primary outcome measure
  • The sample size you would need (a rough power analysis: if d = 0.3 is the expected effect size and you want 80% power at α = .05, how large a sample do you need? The answer is approximately n = 175 per group for an independent samples design)
  • One analysis you would NOT run, even if the data seemed interesting — a commitment to pre-specified analysis scope

Discussion Questions for Seminar

  1. The chapter argues that p-hacking is usually not deliberate fraud. Does this matter morally? Is an unintentional false positive just as damaging to the literature as a deliberate one?

  2. Pre-registration requires knowing your hypothesis in advance. What kinds of research are poorly served by pre-registration? Are there legitimate reasons to run exploratory studies without pre-registration?

  3. The Okafor-Reyes team committed to open data release from the beginning of the study. What are the costs and benefits of open data? Are there circumstances in which open data would be harmful?