Chapter 37 Exercises

How to use these exercises: Work through the parts in order. Part A builds recognition skills, Part B develops analysis, Part C applies concepts to your own domain, Part D requires synthesis across multiple ideas, Part E stretches into advanced territory, and Part M provides interleaved practice that mixes skills from all levels.

For self-study, aim to complete at least Parts A and B. For a course, your instructor will assign specific sections. For the Deep Dive path, do everything.

Part A: Pattern Recognition

These exercises develop the fundamental skill of recognizing survivorship bias across domains.

A1. For each of the following scenarios, identify (a) the selection process, (b) what survived, (c) what was eliminated, and (d) how the elimination biases the conclusions that could be drawn from the survivors.

a) A magazine profiles the "Top 50 Startups of the Decade" and identifies common traits among them: bold vision, willingness to pivot, and aggressive hiring.

b) A fitness influencer posts transformation photos of clients who completed a twelve-week program, showing dramatic results.

c) A university advertises that its graduates earn an average salary of $95,000 within five years of graduation.

d) A documentary about centenarians identifies their shared habits: moderate drinking, strong social connections, and a positive attitude.

e) A historian writes that Roman concrete was superior to modern concrete, based on the fact that Roman structures are still standing after two thousand years.

f) A venture capitalist argues that the best companies are led by founders who "move fast and break things," citing Facebook, Uber, and Airbnb as evidence.

g) A music critic writes that the 1970s was the greatest decade for rock music, citing Led Zeppelin, Pink Floyd, and David Bowie.

h) A financial advisor recommends a mutual fund that has outperformed its benchmark for ten consecutive years.

A2. For each of the following claims, describe the "silent graveyard" -- the evidence that is missing because the selection process destroyed it. How would the claim change if the graveyard's evidence could be included?

a) "This neighborhood is safe -- there have been no reported incidents of violent crime in the past year."

b) "Alternative medicine cured my cancer -- I'm living proof."

c) "Our investment strategy has produced positive returns every year for the past fifteen years."

d) "The ancient Egyptians were the most advanced civilization of the ancient world."

e) "Entrepreneurs who follow their passion are more likely to succeed."

f) "Students who attend elite universities earn more than students who attend non-elite universities."

g) "This drug has been on the market for twenty years with no reported serious side effects."

h) "Traditional building techniques are superior -- just look at how long these old houses have lasted."

A3. Classify each of the following as primarily involving survivorship bias, the streetlight effect (Ch. 35), narrative capture (Ch. 36), or a combination. Justify your classification.

a) A company studies its most successful salespeople to develop a training program, without studying its least successful salespeople.

b) A researcher publishes only the experiments that produced statistically significant results.

c) A journalist writes a compelling profile of a tech founder, focusing on the pivotal moments that led to success and downplaying the role of early investors and market timing.

d) A medical textbook describes the symptoms of a disease based primarily on cases seen in academic hospitals, which serve a non-representative patient population.

e) A history textbook presents the American Revolution through the lens of the founding fathers' correspondence, which survived in archives.

f) A meta-analysis combines fifteen published studies on a supplement's effectiveness, all of which found positive effects.

g) An architect argues that brutalist buildings are ugly by pointing to several notorious examples, ignoring the hundreds of brutalist buildings that are well-loved by their communities.

h) A self-help book draws lessons from the biographies of Abraham Lincoln, Winston Churchill, and Nelson Mandela, presenting resilience as the key to greatness.

A4. Abraham Wald's bomber insight required "inverting" the evidence -- recognizing that the absence of bullet holes in certain areas was more informative than their presence in other areas. For each of the following, describe what "inverting the evidence" would look like.

a) A hospital reports that patients who complete physical therapy have better outcomes than patients who do not.

b) A school district reports that students in its gifted program score higher on standardized tests than students who are not in the program.

c) A country reports that its citizens who hold university degrees have lower unemployment rates than those who do not.

d) A restaurant chain reports that its locations in affluent neighborhoods are more profitable than its locations in low-income neighborhoods.

e) An app developer reports that users who engage with the app daily are more satisfied than users who do not.

f) A pharmaceutical company reports that patients who take their medication as prescribed have fewer hospitalizations.

Part B: Analysis and Explanation

These exercises develop the ability to analyze the structural mechanisms of survivorship bias.

B1. Explain why survivorship bias systematically pushes conclusions in one direction -- toward overconfidence and overoptimism -- rather than introducing random noise. Use at least two examples from the chapter to support your argument.

B2. The chapter distinguishes between the streetlight effect (Ch. 35) and survivorship bias (Ch. 37) as follows: the streetlight effect means we are looking in the wrong place; survivorship bias means the right place has been destroyed. Develop this distinction using the following pairs of examples:

a) The WEIRD problem in psychology (Ch. 35) vs. publication bias in psychology (Ch. 37) b) Valley bias in archaeology (Ch. 35) vs. the loss of perishable-material civilizations (Ch. 37) c) Hot-spot policing data bias (Ch. 35) vs. the disappearance of records from disbanded police units (Ch. 37)

B3. The healthy survivor effect in medicine creates survivorship bias even in well-designed studies. Explain the three mechanisms by which this occurs: a) Self-selection into treatment b) Differential attrition during trials c) Healthy user bias in observational studies

For each, describe a specific scenario and explain how randomization does or does not address the problem.

B4. Fund manager survivorship bias inflates reported average returns by approximately one to two percentage points per year. Calculate the impact of this bias over: a) A ten-year investment horizon (assume a "true" average return of 7% per year) b) A twenty-year investment horizon c) A thirty-year investment horizon

Explain why even a seemingly small annual bias becomes significant over long time periods, and connect this to the lifecycle S-curve concept from Chapter 33.

B5. The chapter argues that survivorship bias in military history creates "a systematic illusion: the strategies that produced victory appear to be better strategies." Develop a detailed analysis of this claim using one of the following examples: a) The Allied strategy in World War II vs. the Axis strategy b) Alexander the Great's military campaigns c) The American strategy in the Revolutionary War

In your analysis, identify what evidence has survived, what evidence has been lost, and how the loss biases our understanding of the conflict.

B6. Explain why meta-analyses -- which are designed to be more reliable than individual studies -- can actually amplify survivorship bias rather than correct it. Under what conditions would a meta-analysis reliably detect and correct for publication bias? Under what conditions would it fail?

Part C: Application to Your Own Domain

These exercises ask you to apply survivorship bias concepts to fields you know personally.

C1. Identify the three most prominent "success stories" in your field or industry. For each one: a) Describe the selection process that made this story visible. b) Estimate the size of the "silent graveyard" -- how many similar attempts failed and are now invisible? c) Identify at least one conclusion commonly drawn from this success story that would change if the graveyard were visible. d) Describe what it would take to actually study the graveyard.

C2. Identify one dataset, body of evidence, or knowledge base in your field that is likely contaminated by survivorship bias. For this dataset: a) What is the selection process that filters the data? b) What type of evidence is systematically removed by the filter? c) How would conclusions drawn from this data change if the missing evidence were included? d) What countermeasure from Section 37.10 would be most effective, and what would it cost to implement?

C3. Write a one-page "survivorship audit" of your own career or educational path. Identify the moments where survivorship bias has shaped your understanding of success in your field. What do you know about the "graveyard" -- the people who followed similar paths and did not succeed? What would you believe differently about your field if you had access to the graveyard's data?

C4. Design a study or investigation in your field that would explicitly seek to overcome survivorship bias. Describe: a) The question you would investigate b) How you would identify and include the "dead" (the non-survivors, the failures, the missing data) c) What practical obstacles you would face d) How your findings might differ from the survivorship-biased conclusions currently accepted in your field

Part D: Synthesis and Cross-Chapter Integration

These exercises require integrating survivorship bias with concepts from earlier chapters.

D1. Survivorship Bias + Feedback Loops (Ch. 2) Explain how survivorship bias can create a feedback loop. Use the following example: a company studies its successful products to identify what made them succeed, then applies those lessons to new products, then studies the new successes to refine its understanding further. How does survivorship bias operate in each cycle of this loop? Does the bias amplify or diminish over time? Why?

D2. Survivorship Bias + Signal and Noise (Ch. 6) Chapter 6 discussed the challenge of separating signal from noise in data. Survivorship bias adds a new dimension: the signal itself may be an artifact of the selection process. Develop this idea using the example of fund manager performance. What appears to be "signal" (consistent outperformance)? What is the "noise" (random variation)? How does survivorship bias make noise look like signal?

D3. Survivorship Bias + Narrative Capture (Ch. 36) Chapter 36 examined how compelling narratives hijack reasoning. Explain how survivorship bias and narrative capture reinforce each other. Why are success stories -- which are survivorship-biased by definition -- also the most narratively compelling? How does narrative structure make survivorship bias harder to detect?

D4. Survivorship Bias + Base Rates (Ch. 10) Chapter 10 examined base rate neglect. Explain how survivorship bias is a mechanism that produces base rate neglect. Specifically, explain how the elimination of failures from the sample distorts the perceived base rate of success. Use a concrete example involving startup entrepreneurship.

D5. Survivorship Bias + Lifecycle S-Curve (Ch. 33) The lifecycle S-curve describes the growth, maturation, and decline of systems. How does survivorship bias interact with the position on the S-curve? Argue that survivorship bias is more severe for older systems (more time for selection to operate) and less severe for younger ones. Then consider a counterargument: in what situations might survivorship bias be more severe for young systems?

D6. Survivorship Bias + Streetlight Effect (Ch. 35) The chapter argues that survivorship bias and the streetlight effect are "related but structurally distinct." Map out the structural differences using a 2x2 matrix with axes of (1) whether the missing evidence exists but is not being observed (streetlight) vs. has been destroyed (survivorship), and (2) whether the bias is correctable by redirecting attention vs. requires reconstructing lost evidence. Populate each cell with examples from Chapters 35 and 37.

Part E: Advanced and Extended Problems

These exercises push beyond the chapter's explicit content into deeper territory.

E1. The Survivorship Bias of Survivorship Bias Examples The chapter uses a set of well-known examples to illustrate survivorship bias: Wald's bombers, business success literature, ancient buildings, medieval music. Is this set of examples itself subject to survivorship bias? That is, are these the examples that "survived" in the intellectual discourse about survivorship bias because they are vivid and memorable, while other, potentially more important examples have been forgotten? Identify at least two domains where survivorship bias operates powerfully but is rarely discussed. Why have these examples been filtered out of the standard discourse?

E2. Evolutionary Survivorship Bias Darwin's theory of natural selection is, in a sense, a theory of survivorship bias applied to biology: the organisms we observe are the survivors of billions of years of selection, and the organisms that did not survive are invisible. Does this mean that all evolutionary reasoning is contaminated by survivorship bias? Or is natural selection a case where survivorship bias is the phenomenon itself rather than a distortion of the phenomenon? Develop a careful argument.

E3. Survivorship Bias in the Intellectual Canon The ideas, theories, and frameworks that constitute "knowledge" in any discipline have survived a selection process: peer review, replication, cultural transmission, institutional adoption. Is the intellectual canon subject to survivorship bias? That is, have ideas survived because they were true, or because they were memorable, institutionally convenient, or politically useful? Develop your argument using examples from at least two disciplines.

E4. Designing a Survivorship-Bias-Free Institution If survivorship bias is structural -- built into the selection mechanisms of every domain -- what would an institution designed to minimize survivorship bias look like? Design a hypothetical research institute, investment firm, or school that incorporates structural countermeasures against survivorship bias. Describe its data collection practices, its incentive structures, its decision-making processes, and how it would differ from conventional institutions.

E5. The Ethics of Silent Evidence Survivorship bias has ethical dimensions: the people whose evidence is "silent" (the dead patients, the bankrupt entrepreneurs, the destroyed civilizations) often suffered the worst outcomes. Their silence is not chosen. It is imposed by the process that destroyed them. Develop an ethical argument about the obligations of researchers, institutions, and policymakers toward the silent -- those whose evidence has been destroyed by the processes being studied.

Part M: Mixed Practice (Interleaved)

These exercises deliberately mix concepts from different sections and chapters to build flexible retrieval.

M1. A pharmaceutical company advertises that its cholesterol drug reduces the risk of heart attack by 40%, based on a clinical trial in which participants who completed the full two-year protocol had significantly better outcomes than the placebo group. Identify at least three distinct biases that might be operating (draw on Chapters 10, 35, 36, and 37) and explain how each affects the reliability of the 40% claim.

M2. A management consultant tells you: "I've studied the hundred fastest-growing companies of the past decade and identified six traits they all share. If your company adopts these traits, you'll grow faster." Without rejecting the consultant's findings outright, formulate five specific questions you would ask to evaluate this claim, drawing on concepts from at least three different chapters.

M3. You are advising a government health ministry that wants to evaluate whether a public health intervention (free gym memberships for all citizens over 50) is effective. The ministry proposes to compare health outcomes for people who used the gym memberships vs. people who did not. Identify the survivorship bias problem in this design, propose an alternative design that addresses it, and explain what additional biases (from Chapters 10, 35, or 36) might still affect the alternative design.

M4. A Wikipedia article about ancient Roman engineering states: "Roman engineering was far superior to modern engineering, as evidenced by the fact that Roman bridges, aqueducts, and roads are still in use after two thousand years, while modern infrastructure often requires replacement after just a few decades." Write a response (200-300 words) that corrects this claim using concepts from this chapter and at least one other chapter.

M5. You are reviewing a meta-analysis that combines twenty-three published studies examining whether a particular teaching method improves student learning. All twenty-three studies report positive effects. The meta-analysis concludes that the evidence for the teaching method is "robust and consistent." Drawing on the file drawer problem, base rate thinking, and the streetlight effect, compose a methodological critique. What additional information would you need to determine whether the meta-analysis's conclusion is reliable?

M6. A friend says: "I've been investing in the stock market for fifteen years, and my portfolio has beaten the S&P 500 in twelve of those years. That proves I have genuine investing skill." Formulate a response that incorporates survivorship bias, base rates (Ch. 10), and the distinction between signal and noise (Ch. 6). Be precise about what additional information you would need to evaluate your friend's claim.

M7. Consider the following argument: "Survivorship bias is itself a form of the streetlight effect -- we study the survivors because they are visible, just as the drunk searches under the streetlight because the light is there." Evaluate this argument. Is survivorship bias a special case of the streetlight effect, or is it structurally distinct? What does the distinction (or lack thereof) tell us about the relationship between these two cognitive biases?