Case Study: When the Checklist Reveals a Sick Organization
The Scenario
Consider a composite organization — an amalgam of patterns documented throughout this book — that we'll call Meridian Research Institute (composite example created for this text). Meridian is a well-funded research organization in a respected field. It employs 200 researchers, publishes in top journals, and has significant influence on policy.
An employee applies the Epistemic Health Checklist to Meridian and produces the following profile:
| Dimension | Score | Evidence |
|---|---|---|
| 1. Dissent tolerance | 3 | Two researchers who challenged a core methodology were reassigned; junior researchers self-censor |
| 2. Replication culture | 2 | No resources allocated for replication; novel findings are the exclusive publication priority |
| 3. Incentive alignment | 2 | Funding depends on producing results that support the funder's policy position; negative results are buried |
| 4. Measurement validity | 4 | Some metrics are validated; the core outcome metric is a proxy with known limitations that are not discussed |
| 5. Outsider access | 3 | External researchers can submit papers but face "not our methodology" objections; no external review board |
| 6. Correction speed | 3 | One known error took 8 years to correct; no formal correction process |
| 7. History awareness | 2 | The institute's founding narrative omits its early methodological controversies |
| 8. Claim falsifiability | 4 | Most claims are testable; one core assumption is treated as axiomatic |
| 9. Method diversity | 3 | A single methodology dominates; researchers trained in alternative methods have left |
| 10. Process transparency | 3 | Data are available on request but rarely requested; methods sections omit key analytical choices |
| Average | 2.9 | Critical epistemic health failure |
Interpreting the Profile
Meridian's score of 2.9 places it in the "critical" range — the structures actively protect error and suppress correction. The profile reveals several specific vulnerabilities:
The suppression cluster (D1 + D5 + D10). Low dissent tolerance, low outsider access, and low transparency combine to create an environment where errors cannot be detected from inside (dissenters are punished) or outside (outsiders are excluded, processes are opaque).
The incentive-evidence cluster (D2 + D3). No replication culture combined with misaligned incentives means that findings are produced once, never checked, and the system rewards confirmation of the funder's preferred outcome.
The methodology trap (D9 + D8). Low method diversity combined with one unfalsifiable core assumption means the organization cannot see evidence that its dominant method misses, and cannot question the foundation on which all its work rests.
What Would the Framework Predict?
Based on the failure mode framework, Meridian's profile predicts:
- Its published findings will have a high rate of non-replication — because no one checks, and the incentives reward producing positive results regardless of their robustness
- Errors will persist for years or decades — because the combination of low dissent tolerance and low outsider access means that no one with the knowledge and standing to challenge errors has an incentive or an opportunity to do so
- When correction eventually happens, it will be driven by external crisis — because the internal correction mechanisms are suppressed
- The organization will resist correction — because the correction will threaten the funding relationship, the methodological identity, and the careers of senior researchers
- After correction, the organization will sanitize its history — because low history awareness is already part of the pattern
What Can Be Done?
The checklist identifies specific dimensions that could be targeted for improvement:
Quick wins (structural changes that face modest resistance): - Establish an external advisory board with genuine authority (improves D5, D1) - Implement pre-registration for studies (improves D2, D10) - Require full disclosure of analytical choices in methods sections (improves D10)
Medium-term reforms (structural changes that face significant resistance): - Diversify funding sources to reduce dependence on a single funder (improves D3) - Hire researchers trained in alternative methodologies (improves D9) - Create a formal correction process with published timelines (improves D6)
Hard structural changes (likely to face intense resistance): - Challenge the unfalsifiable core assumption (D8) — this threatens the foundation of the organization's work - Reform the incentive structure so that negative results and replications are valued (D2, D3) — this threatens publication records and funding - Honestly acknowledge the organization's error history (D7) — this threatens institutional reputation
The Political Reality
The employee who conducted this assessment faces a dilemma: the assessment reveals genuine structural problems, but sharing it risks their position within the organization (which scored 3/10 on dissent tolerance). This is the outsider problem (Chapter 18) operating within an organization — the person with the diagnosis is structurally unable to deliver it.
This dilemma is itself predicted by the checklist: an organization with low dissent tolerance will suppress exactly the kind of internal critique that the checklist represents. The tool works best when it is used by people with enough structural protection (tenure, seniority, external position) to survive delivering uncomfortable findings.
Analysis Questions
1. The employee who conducted this assessment has identified genuine problems but faces career risk in sharing the results. Using the practical strategies from Chapter 33 (How to Disagree Productively — preview), design a strategy for presenting the checklist results that maximizes the probability of being heard while minimizing career risk.
2. Meridian's lowest scores are on dimensions 1 (dissent tolerance), 2 (replication culture), and 7 (history awareness). If you could improve only one of these by 3 points, which would produce the largest cascade effect on the other dimensions? Explain the causal chain.
3. Apply the Correction Speed Model (Chapter 22) to predict how long it would take Meridian to correct a known error, given its health profile. What variables from the model are captured in the checklist, and what variables would you need additional information to assess?
4. Compare Meridian's profile to nutrition science's profile in the worked example. The scores are similar (~3). What structural features do they share? What does this suggest about the generalizability of epistemic health failures across very different types of organizations?