Chapter 37 Quiz: Self-Assessment

Instructions: Answer each question without looking back at the chapter. After completing all questions, check your answers against the key at the bottom. If you score below 70%, revisit the relevant sections before moving on to Chapter 38.

Multiple Choice

Q1. Survivorship bias is best defined as:

a) The tendency to study only successful outcomes because they are more interesting b) The systematic error of drawing conclusions from what survived a selection process while ignoring what did not survive, leading to systematically wrong conclusions about what caused the survival c) The bias that occurs when survivors of a disaster give more vivid testimony than non-survivors d) The tendency to assume that whatever has survived the longest is the best

Q2. Abraham Wald's key insight about the WWII bombers was that:

a) The bombers needed more armor overall to survive enemy fire b) The bullet holes on returning planes showed where armor was needed most c) The areas without bullet holes on returning planes were the areas where a hit was fatal -- because planes hit there did not return d) The engines were the least important part of the plane to armor, since few returning planes showed engine damage

Q3. The fundamental problem with business success literature like Good to Great is that:

a) The research methodology was careless and the data was poorly collected b) It studied only companies that had already succeeded, making it impossible to determine whether the identified traits distinguished winners from losers -- since companies with the same traits that failed were invisible c) The companies it profiled were outliers that could not be compared to typical firms d) The authors had financial conflicts of interest that biased their conclusions

Q4. Survivorship bias in music and cultural memory produces the illusion that:

a) Modern music is technically more sophisticated than historical music b) All historical music was of uniformly high quality, because time has filtered out everything mediocre and preserved only the best c) Musical quality has been steadily declining since the Renaissance d) Ancient civilizations did not produce popular or informal music

Q5. The healthy survivor effect in medicine refers to:

a) The fact that healthier patients are more likely to choose treatments, more likely to complete studies, and more likely to survive -- creating a biased comparison between treated and untreated groups b) The tendency for doctors to prescribe treatments only to patients who are already healthy c) The observation that survivors of serious illness develop stronger immune systems d) The bias in clinical trials caused by deliberately enrolling only healthy patients

Q6. Differential attrition in clinical trials creates survivorship bias because:

a) Patients in the treatment group always drop out faster than patients in the control group b) Sicker patients and those who experience side effects are more likely to leave the study, so the remaining sample overrepresents patients for whom the treatment was tolerable c) Patients who die during the trial are counted as treatment failures d) Clinical trials are too short to capture long-term survivorship effects

Q7. Fund manager survivorship bias operates through which mechanism?

a) Fund managers who perform poorly are fired, and their personal track records are erased b) Funds that underperform significantly are often closed, merged, or dissolved -- and their performance records disappear from the databases used to evaluate fund manager performance c) Financial regulators require that poorly performing funds be hidden from public view d) Investors who lose money in funds refuse to report their losses

Q8. Nassim Nicholas Taleb's concept of "silent evidence" refers to:

a) Evidence that is classified or kept secret by governments b) Evidence that has been destroyed, hidden, or rendered invisible by the very process being studied -- making it structurally impossible to observe c) Evidence from witnesses who refuse to testify d) Statistical data that falls below the threshold of significance

Q9. The file drawer problem in scientific publishing refers to:

a) The tendency for researchers to lose their raw data in disorganized filing systems b) The preferential publication of studies with positive findings, while studies with null or negative results are filed away and never published -- creating a survivorship-biased scientific record c) The problem of data storage capacity in pre-digital research archives d) The tendency for journals to reject studies that use novel methodologies

Q10. Ioannidis's argument that "most published research findings are false" depends critically on:

a) The assumption that most researchers are incompetent b) The mathematical demonstration that under realistic conditions -- small samples, small effects, publication bias, and multiple testing -- the published literature is enriched for false positives because true negatives are filtered out c) The finding that most researchers fabricate their data d) The observation that peer review is insufficiently rigorous

Q11. Base rate thinking is a countermeasure to survivorship bias because:

a) It provides a formula for calculating the exact magnitude of the bias b) It forces you to consider the full population -- survivors and non-survivors -- rather than drawing conclusions from the survivors alone, by anchoring on the overall rate of success or failure in the relevant reference class c) It eliminates the need to study failures by providing a statistical correction factor d) It replaces individual success stories with aggregate data, which is always unbiased

Q12. The "outside view" (Kahneman and Tversky) counteracts survivorship bias by:

a) Requiring that all decisions be made by external consultants who are not subject to cognitive biases b) Evaluating a project or prediction by examining how similar projects or predictions have performed historically -- including the failures -- rather than focusing on the specific features of the current case c) Using satellite imagery and remote sensing to detect evidence that ground-level observation would miss d) Requiring that researchers publish their data publicly so that others can check for survivorship bias

Q13. Pre-registration of scientific studies addresses publication bias by:

a) Requiring that all studies be published regardless of their results, by having journals commit to publication based on the quality of the research design before data is collected b) Allowing researchers to register their hypotheses after seeing the results, which corrects for hindsight bias c) Creating a public database of all studies that have been conducted, including those that were never submitted for publication d) Requiring that researchers disclose their funding sources before their studies can be published

Q14. The chapter argues that the structural difference between the streetlight effect and survivorship bias is:

a) The streetlight effect operates in science while survivorship bias operates in business b) The streetlight effect means we are looking in the wrong place, while survivorship bias means the right place has been destroyed -- the evidence no longer exists c) The streetlight effect is caused by institutional incentives while survivorship bias is caused by chance d) The streetlight effect is correctable while survivorship bias is not

Q15. The threshold concept "The Evidence Destroys Itself" means:

a) Physical evidence degrades over time due to natural entropy b) In many domains, the process of success or survival systematically eliminates the evidence of failure -- making it structurally impossible to learn from failure unless you deliberately seek out the dead c) Researchers who find negative results destroy their evidence to protect their careers d) Evidence is always less reliable than theory because it is subject to measurement error

Short Answer

Q16. In one sentence, explain why comparing medieval music to modern music is an unfair comparison biased by survivorship.

Q17. Name three domains discussed in the chapter where survivorship bias operates and, for each, identify the specific selection mechanism that filters out the non-survivors.

Q18. Explain the connection between survivorship bias and base rate neglect (Ch. 10) in one to two sentences.

Q19. The chapter claims that survivorship bias systematically makes five things appear different from what they actually are. Name at least three of the five.

Q20. Explain why a meta-analysis of published studies can amplify rather than correct survivorship bias.

True or False

Q21. Survivorship bias is primarily caused by deliberate fraud or data manipulation.

Q22. Abraham Wald recommended armoring the areas of the bombers that showed the most bullet holes.

Q23. The file drawer problem affects only psychology and social science research, not medical or physical science research.

Q24. Survivorship-bias-free databases in finance consistently show weaker evidence of fund manager skill than databases that include only surviving funds.

Q25. The healthy survivor effect can contaminate even randomized controlled trials through the mechanism of differential attrition.

Answer Key

Q1. b) -- Survivorship bias is the systematic error of drawing conclusions from what survived a selection process while ignoring what did not survive. (Section 37.1)

Q2. c) -- Wald recognized that the areas without bullet holes on returning planes were the areas where a hit was fatal, because planes hit in those areas did not return. (Section 37.1)

Q3. b) -- The fundamental problem is that studying only successful companies makes it impossible to determine whether the identified traits are actually causes of success, since failed companies with the same traits are invisible. (Section 37.2)

Q4. b) -- Survivorship bias in cultural memory filters out mediocre historical work, making the surviving masterpieces appear to be the norm rather than the exception. (Section 37.3)

Q5. a) -- The healthy survivor effect occurs because healthier, wealthier, and more health-conscious patients are more likely to choose treatments and survive long enough to be studied, contaminating the comparison. (Section 37.5)

Q6. b) -- Differential attrition removes sicker patients and those with side effects from the sample, leaving a biased remainder that overrepresents patients for whom the treatment was tolerable. (Section 37.5)

Q7. b) -- Funds that underperform are closed, merged, or dissolved, and their records disappear from performance databases, inflating the apparent average performance of surviving funds. (Section 37.7)

Q8. b) -- Silent evidence is evidence that has been destroyed, hidden, or rendered invisible by the very process being studied. (Section 37.8)

Q9. b) -- The file drawer problem is the preferential publication of positive findings and the suppression of null results, creating a biased scientific record. (Section 37.9)

Q10. b) -- Ioannidis's argument depends on the mathematical demonstration that publication bias enriches the literature for false positives under realistic research conditions. (Section 37.9)

Q11. b) -- Base rate thinking forces consideration of the full population, including non-survivors, by anchoring on the overall rate of success or failure. (Section 37.10)

Q12. b) -- The outside view examines historical outcomes for similar projects, including failures, rather than focusing on the specific features of the current case. (Section 37.10)

Q13. a) -- Pre-registration eliminates the file drawer by having journals commit to publication based on research design quality, regardless of results. (Section 37.10)

Q14. b) -- The streetlight effect means we are not looking in the right place; survivorship bias means the right place has been destroyed. (Section 37.8)

Q15. b) -- The process of survival systematically eliminates the evidence of failure, making it structurally impossible to learn from failure without deliberate effort. (Section 37.11)

Q16. Sample answer: We compare the curated highlights of centuries (only the best medieval music survived) against the full, unfiltered range of everything being produced today (including the terrible), so the past always appears better because time has silently removed its mediocrity. (Section 37.3)

Q17. Sample answer: Business (bankruptcy eliminates failed companies from study), music (cultural forgetting destroys mediocre compositions), finance (fund closure removes poor performers from databases). Other valid answers: architecture (structural failure removes bad buildings), medicine (death and dropout remove sick patients), military history (conquest destroys the defeated civilization's records), science (editorial rejection removes null results). (Sections 37.2-37.9)

Q18. Sample answer: Survivorship bias hides the denominator (the failures) from view, making the observed success rate appear much higher than the true base rate; without seeing the failures, we cannot correctly estimate the probability of success. (Sections 37.2, 37.10)

Q19. The five: (1) Success looks easier than it is, (2) winning strategies look more reliable than they are, (3) talent looks more common than it is, (4) risk looks smaller than it is, (5) the past looks better than the present. Any three is sufficient. (Section 37.11)

Q20. A meta-analysis aggregates published studies, and if the underlying studies are biased by publication bias (positive results preferentially published, null results filed away), the meta-analysis inherits and amplifies the bias -- it is combining only the "survivors" of the publication filter, making the effect appear larger and more consistent than it is. (Section 37.9)

Q21. False. Survivorship bias is a structural feature of selection processes, not the result of fraud. It operates even when all participants are honest and competent. (Section 37.1, 37.8)

Q22. False. Wald recommended the opposite: armoring the areas that showed the fewest bullet holes on returning planes, because those were the areas where damage was fatal. (Section 37.1)

Q23. False. The file drawer problem affects all scientific disciplines, including medicine and physical sciences. (Section 37.9)

Q24. True. Survivorship-bias-free databases consistently show weaker evidence of fund manager skill than biased databases. (Section 37.7, 37.10)

Q25. True. Differential attrition -- sicker patients dropping out before study completion -- introduces survivorship bias even into randomized controlled trials. (Section 37.5)