Key Takeaways: Survivorship Bias at Scale

The Big Idea

Survivorship bias is the systematic error of drawing conclusions from evidence that survived a selection process while ignoring what was filtered out. Because the missing evidence is invisible, the bias is extraordinarily difficult to detect — and it always makes things look better than they are.

Core Concepts

Wald's Insight

The evidence you DON'T see is often more important than the evidence you do. The bullet holes on surviving bombers showed where planes could be hit and survive — not where they were most vulnerable.

Wald's Five-Step Template

  1. Identify the evidence in front of you
  2. Identify the selection process (what filter produced this evidence?)
  3. Ask what the filter excluded
  4. Determine whether the filter correlates with your variable of interest
  5. Adjust conclusions accordingly

Publication Bias

  • Published literature systematically overrepresents positive results
  • Null results are filed away, creating a biased evidence base
  • The antidepressant case: 94% positive in published literature vs. ~51% in full evidence
  • Funnel plots can detect the asymmetry visually

The Denominator Problem

Without knowing the failure rate, you cannot evaluate the success rate. Studying only winners tells you about survival, not about causes of success.

Cross-Domain Examples

Domain What Survived What Didn't Distortion
WWII bombers Planes that returned Planes shot down Armored wrong areas
Business literature Successful companies Failed companies with same traits False "success factors"
Ancient history Literate civilizations Oral cultures Biased theories of development
Medical research Published positive studies Filed null studies Inflated treatment effects
Mutual funds Surviving funds Closed/merged funds Inflated performance data
Startup mythology Unicorn founders Millions of identical failed founders False "success recipes"

Practical Strategies

  1. Always ask about the denominator
  2. Seek pre-registered evidence
  3. Look for systematic reviews that include unpublished data
  4. Study failures, not just successes
  5. Build a mental model of the selection process

Epistemic Audit — Chapter 5 Addition

After this chapter, your audit should include: what evidence didn't survive in your field, whether publication bias exists, what the denominator is for key claims, and what selection processes filter evidence before it reaches you.

What's Coming Next

Chapter 6: The Plausible Story Problem — when narrative coherence substitutes for evidence.

Quick Reference: Wald's Question

For ANY body of evidence, ask:
"What is this a survivor of?"
"What evidence was filtered out?"
"Would the filtered evidence change my conclusion?"