Chapter 9 Exercises: Survivorship Bias — The Most Dangerous Lie Statistics Tell

Level 1: Recall and Comprehension

1.1 Explain Abraham Wald's airplane problem in your own words. What data did the military have? What conclusion did they initially draw? Why was Wald's alternative conclusion correct?

1.2 Define survivorship bias. What are the three components of the mechanism? Give a real-world example of each component.

1.3 Explain the "invisible graveyard" metaphor. Why is the graveyard the most important data in survivorship bias situations?

1.4 What is the selection effect in the context of elite university outcomes? How does the Dale-Krueger research address it?

1.5 How does survivorship bias distort mutual fund performance data? What is the approximate magnitude of the bias in terms of percentage points per year?

1.6 List five specific advantages that survivorship bias hides in typical content creator success stories.

1.7 What are the five tools for identifying and correcting survivorship bias described in this chapter? Give a one-sentence description of each.

Level 2: Application

2.1 You are reading a book by a famous entrepreneur who built a billion-dollar company. The book recommends: (a) working 80+ hours per week, (b) ignoring market research and trusting your instincts, (c) turning down early acquisition offers. Identify which of these pieces of advice might be contaminated by survivorship bias and explain why. For each, describe what the "invisible graveyard" of founders who followed the same advice would look like.

2.2 Nadia follows Tyler Ash's advice for 11 weeks: daily posting, narrow niche, reply to all comments. Her channel doesn't grow meaningfully. She's frustrated. Write a 300-word explanation for Nadia, using survivorship bias concepts, of why Tyler's advice might be simultaneously true (about his experience) and misleading (as general advice for her).

2.3 A sports commentator says: "All the great quarterbacks in NFL history had strong-armed coaches who were demanding and critical. Clearly, demanding coaching produces greatness." Apply survivorship bias analysis. What data is the commentator looking at? What data is missing? What question would you need to answer before accepting this conclusion?

2.4 You find a statistic that says "90% of millionaires read at least one non-fiction book per month." You are about to start a daily reading habit because you want to be successful. Identify the survivorship bias problem with using this statistic to motivate your habit. What additional information would you need?

2.5 Analyze the Lindy Effect (mentioned in the chapter's Myth vs. Reality box): the idea that things that have survived for a long time are likely to survive much longer. This is explicitly based on survivorship. Does the Lindy Effect use survivorship bias correctly or incorrectly? Is survivorship bias always an error, or can it sometimes provide valid information?

2.6 The chapter mentions that investment databases that don't correct for survivorship bias show funds performing better than databases that do correct for it. Describe, step by step, the mechanism that produces this difference. If you were an index fund company trying to show the advantage of passive investing, how would you use this finding?

Level 3: Analysis

3.1 Marcus is reading a book about chess champions and notices that almost all of them describe their training as involving enormous amounts of deliberate practice — studying games, working with coaches, solving tactical puzzles for hours daily. He concludes that deliberate practice is the key to chess mastery. Analyze this conclusion for survivorship bias. What data is Marcus looking at? What data is missing? What would a properly designed study of chess skill acquisition look like?

3.2 Compare survivorship bias in three domains: (a) startup advice, (b) health and longevity advice from centenarians, (c) investment track records. For each domain: What is the selection mechanism? Who are the survivors? What data is in the invisible graveyard? What is the most dangerous false conclusion survivorship bias produces in each domain?

3.3 The chapter argues that survivorship bias is "the most dangerous lie statistics tell" because it uses true facts to mislead. Develop this argument more fully. What specifically makes survivorship bias more dangerous than, say, outright false statistics? In what sense is a true fact from a survivorship-biased sample misleading? Generate two examples where the true facts are even more misleading than a false claim would be.

3.4 Priya is job searching and notices that most of the people she admires in her target industry went to a small set of prestigious universities. She is considering whether to get a second degree from one of those universities to improve her prospects. Apply survivorship bias analysis to this career decision. What might explain the pattern she's observing other than the university being causally responsible for their success? What research should she consult? What alternative strategies might avoid the survivorship trap?

Level 4: Synthesis and Evaluation

4.1 Design a study that could actually measure whether Tyler Ash's content creation advice is causally responsible for channel growth — as opposed to being correlated with growth in a survivorship-biased sample. Your study should include: (a) the research question, precisely stated; (b) how you would recruit participants; (c) what the control condition would look like; (d) what outcome measures you would use; (e) how long the study would run; (f) what statistical analysis you would use. Evaluate the feasibility and ethics of your design.

4.2 Develop a framework for evaluating advice from successful people that explicitly accounts for survivorship bias. Your framework should include: (a) questions to ask about the advice-giver's reference class; (b) questions to ask about the advice-giver's unique advantages; (c) methods for finding information about people who followed similar advice and failed; (d) a method for weighting the advice appropriately given survivorship. Apply this framework to one piece of common life advice you have received.

4.3 The chapter describes survivorship bias in mutual funds showing 1-3 percentage points of apparent performance inflation. Evaluate this magnitude carefully: In a market where genuine skill might produce 1-2 percentage points above benchmark returns, what does a 1-3 point survivorship bias mean for the interpretation of any fund's track record? How long a track record, accounting for survivorship bias, would you need before concluding a manager has genuine skill rather than lucky survival?

4.4 The Abraham Wald story has become almost universally known in data science and statistics education. Evaluate it as a teaching tool: What does it teach well? What might it teach poorly or incompletely? What aspects of survivorship bias does the airplane metaphor not fully capture? Design a different metaphor or example that captures the aspects the airplane story misses.

Level 5: Creative and Personal Application

5.1 Write the book that Tyler Ash should have written — or at least the introduction to it. This version of the book would honestly grapple with survivorship bias: acknowledging the selection mechanism, describing the advantages Tyler had that many aspiring creators don't, giving a realistic estimate of the fraction of creators who follow his advice and reach his level, and offering advice that's calibrated for the actual probability distribution of outcomes. Write 500-700 words as this more honest book's introduction.

5.2 Map your "invisible graveyard." Think of a domain where you are seeking to succeed — academic, creative, professional, athletic. Identify: Who are the visible survivors (the people you look to for inspiration and advice)? Who are in the invisible graveyard (people who tried and failed, or who tried the same path and ended up somewhere different)? What can you learn from the graveyard that you can't learn from the survivors?

5.3 Find a piece of advice you've received or a strategy you've adopted based on observing someone else's success. Apply the survivorship bias toolkit from this chapter: Ask about the denominator. Look for hidden advantages. Try to find information about the non-survivors. Assess whether the advice is based on a survivorship-filtered sample. Write a one-page analysis of what you find.

5.4 Abraham Wald's insight was that the most important data was missing from the dataset entirely. Think of a domain in your own life — academic performance, creative work, a relationship, a skill you're developing — where the most important information is in data you're not collecting or not seeing. What is the "invisible graveyard" in your own experience? How would you start collecting it?

5.5 Write a short story (500-700 words) from the perspective of a content creator who tried Tyler Ash's advice for two years and built a channel with 8,000 subscribers — good but not exceptional. The story should be written after they read Tyler Ash's book. What do they make of the gap between the advice and their outcome? What do they wish the book had said? How do they plan to proceed? The story should demonstrate a sophisticated understanding of survivorship bias without ever using the term.