Case Study: The Open Science Movement — A Correction in Progress

Overview

The Open Science movement in psychology is the most ambitious attempt at institutional self-correction in modern social science. This case study evaluates its progress, achievements, and limitations using the correction frameworks from Part III.

The Reform Architecture

The Center for Open Science (COS)

Founded by Brian Nosek in 2013, COS provides infrastructure for Open Science practices: the Open Science Framework (OSF) for pre-registration and data sharing, support for registered reports, and coordination of large-scale replication projects. COS is notable for being funded primarily by philanthropic foundations (Arnold Ventures, Laura and John Arnold Foundation), independent of the academic institutions whose practices it aims to reform.

Key Reform Components

Evaluating the Correction

By the Markers of Genuine Correction (Chapter 19)

New training curricula? Yes. Graduate programs increasingly teach Open Science methods. New textbooks incorporate the replication crisis and reformed methods. This is the strongest indicator of genuine correction — it ensures the next generation is trained differently.

New hiring criteria? Partially. Some departments explicitly value methodological rigor and Open Science practices in hiring. Others continue to prioritize publication count and journal prestige. The transition is uneven.

Changes persist after crisis fades? Appears yes. Pre-registration rates continue to increase. Registered reports continue to grow. The reforms show no sign of reverting. However, the crisis is still relatively recent (~12 years), and generational forgetting (Chapter 19) typically operates on a 20-30 year timescale.

Former defenders acknowledge failure? Some. Several senior social psychologists have publicly acknowledged that the old practices were problematic. Others continue to defend their work or argue that the replication movement is overstated. The field is split, with the split largely (though not entirely) along generational lines.

Correction extends to adjacent areas? Yes. Open Science practices are spreading to medicine, political science, economics, education, and other fields. This is one of the most impressive aspects of psychology's response — it has produced a correction framework that other fields are adopting.

By the Overcorrection Diagnostic (Chapter 21)

Origin Test: The reforms were established in the wake of the replication crisis. ✓ (This doesn't prove overcorrection, but it's a risk factor.)

Mirror Test: Some reforms are approximate opposites of the previous position: "publish anything novel and positive" → "only publish pre-registered confirmatory studies." The mirror risk exists but is not fully realized — most journals accept both pre-registered and non-pre-registered work.

Invisible Cost Test: The costs of the reforms (slower research, chilling effect on exploration, methodological conservatism) are discussed within the field but are not as prominently featured as the costs of the old practices. This is a moderate risk factor.

Independent Evidence Test: Pre-registration and open data would be good practices regardless of the replication crisis — they improve research quality independently of the crisis. This suggests the reforms are not purely reactive.

Accommodation Test: The reforms acknowledge the value of exploratory research (some journals distinguish between confirmatory and exploratory pre-registrations). But the cultural emphasis is heavily on confirmation over exploration.

Overall assessment: The reforms are genuine, not cosmetic. There are moderate overcorrection risks in specific areas (chilling effect on exploratory research, methodological conservatism) but the overall trajectory is toward calibrated correction rather than pendulum overcorrection.

The Unfinished Business

The Incentive Problem Remains

Open Science reforms address the outputs of bad incentives (QRPs) more than the incentives themselves. The fundamental structure — publication count as the metric of academic success, journal prestige as the arbiter of quality — remains largely intact. Pre-registration makes it harder to p-hack, but it doesn't change the fact that researchers who publish more papers in prestigious journals get the best jobs. Until the incentive structure changes, the pressure toward questionable practices will reassert itself in new forms that the current reforms don't anticipate.

The WEIRD Problem Remains

Open Science reforms address methodological quality within existing research populations. They do not address the geographic and cultural bias of the research enterprise. A pre-registered, high-powered, open-data study conducted entirely on American undergraduates is methodologically stronger than an un-registered, low-powered study — but it still may not generalize to the majority of humanity.

The Institutional Memory Question

How will psychology tell the story of the replication crisis in 20 years? Will the messy version be preserved — the decades of known problems, the researchers who were marginalized for raising concerns, the role of external embarrassment in driving reform? Or will the revision myth smooth it into "psychology discovered its problems and self-corrected"?

The answer will determine whether the reforms are sustained or whether generational forgetting allows the same dynamics to reassert themselves in a new form.

Analysis Questions

1. Evaluate the Open Science movement as a case of "genuine correction" using the five markers from Chapter 19. On which markers does it score highest? Where is it weakest?

2. The chapter notes that the incentive structure remains largely unchanged despite the reforms. Design an incentive reform that would complement the Open Science reforms. What would it change? Who would resist it?

3. Brian Nosek and the Center for Open Science are independent of the academic institutions they aim to reform. Apply the outsider framework (Chapter 18): what advantages does this independence provide? What limitations?

4. The Open Science movement is spreading to other fields. Apply the Correction Speed Model: which fields are most likely to adopt Open Science practices (high outsider access, high alternative availability) and which are least likely (high switching cost, high defender power)?

5. Using the revision myth diagnostic (Chapter 20, seven questions), evaluate how the replication crisis is currently being told. Is the narrative already being sanitized? What would a deliberately "messy" institutional history of the crisis look like?

Key Takeaway

Psychology's Open Science movement is the best contemporary example of genuine institutional self-correction in any field. It has changed training, publication, and methodology in ways that persist and are spreading. But the correction is incomplete: the incentive structures that produced the crisis remain largely intact, the WEIRD problem is unaddressed, and the revision myth is already forming. Whether the reforms are sustained — or whether generational forgetting allows the same dynamics to reassert — will be determined by whether the field maintains its current institutional vigilance or relaxes into the comforting narrative that the problem has been solved.