Key Takeaways: How Incentive Structures Manufacture Error
The Big Idea
The infrastructure of knowledge production — funding, research, evaluation, publication, and dissemination — has systematic pro-error biases built into its business model. Honest individuals responding rationally to misaligned incentives collectively produce catastrophic errors. The problem is structural, not moral.
Core Concepts
Misalignment vs. Corruption
- Corruption: Individuals deliberately act against the interests they serve (rare, detectable, addressable through enforcement)
- Misalignment: Honest individuals, acting rationally within the incentive structure, collectively produce harmful outcomes (common, invisible, addressable only through structural redesign)
The Five-Stage Incentive Map
- Funding — Who pays? What do they want?
- Research — What gets studied? What's rewarded?
- Evaluation — Who reviews? What are their incentives?
- Publication — What gets published? What's the selection criterion?
- Dissemination — How does it reach the public? What drives selection?
The Five-Question Incentive Audit
- Who funds the knowledge production and what do they want?
- Who produces it and what are they rewarded for?
- Who evaluates it and what are their incentives?
- Who disseminates it and what drives their selection?
- Who bears the cost when the knowledge is wrong?
Key Cases
| Case | Misalignment | Consequence |
|---|---|---|
| Challenger | Schedule > safety at every level | 7 deaths |
| Rating agencies | Issuer-pays model → systematic overrating | 2008 financial crisis |
| Pharmaceutical trials | Company-funded → efficacy overestimation | Millions of patients undertreated |
| Tobacco/"Doubt" | Industry-funded → manufactured uncertainty | Decades of delayed regulation, millions of deaths |
| Opioids | Commercial incentive → minimized addiction risk | ~500,000 overdose deaths |
Epistemic Audit — Chapter 11 Addition
After this chapter: map the five-stage incentive structure, identify misalignments, assess who bears the cost of error.
What's Coming Next
Chapter 12: Precision Without Accuracy — the seduction of exact numbers that are exactly wrong.
Quick Reference:
"Show me the incentives and I will show you the outcome." — Charlie Munger
WHO FUNDS IT? → WHO BENEFITS FROM SPECIFIC CONCLUSIONS?
WHO PRODUCES IT? → WHAT ARE THEY REWARDED FOR?
WHO BEARS THE COST WHEN IT'S WRONG? → (Usually NOT the people who produced it)