Exercises: Survivorship Bias at Scale

Difficulty Guide: - ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Challenging | ⭐⭐⭐⭐ Advanced/Research

Part A: Conceptual Understanding ⭐

A.1. Explain Abraham Wald's insight in your own words. Why did the military's initial conclusion (armor the fuselage) represent survivorship bias?

A.2. Define the "file drawer problem." Why is it a form of survivorship bias?

A.3. What is the "denominator problem" in the context of survivorship bias? Give an example from a field not discussed in the chapter.

A.4. Explain why survivorship bias always makes things "look better than they are." What structural feature of the bias produces this directional effect?

A.5. The chapter describes five steps in Wald's diagnostic template. List them and explain why step 4 (determining whether the filter correlates with your variable of interest) is the critical step.

A.6. What is a funnel plot, and how does it help detect publication bias?

Part B: Applied Analysis ⭐⭐

B.1. Choose a "success story" from your field (a successful company, practitioner, intervention, or approach). Apply Wald's five-step template: identify the evidence, the selection process, what was excluded, whether the filter correlates with your variable, and how conclusions should be adjusted.

B.2. The chapter discusses the antidepressant publication bias case. Find another example of publication bias in medicine (or another field) and analyze it using the same framework. What percentage of evidence was visible? What was hidden?

B.3. Analyze a self-help or business advice book using the survivorship bias framework. What selection process produced the book's evidence? What evidence is missing?

B.4. Compare survivorship bias in mutual fund advertising with survivorship bias in university ranking systems. What structural similarities exist?

B.5. The chapter identifies survivorship bias in our understanding of ancient history. Choose a specific historical claim and analyze what evidence might be missing due to the literacy, building, or "great man" biases.

Part C: Research Design Challenges ⭐⭐–⭐⭐⭐

C.1. Design a study of business success that avoids survivorship bias. What would the study look like? What data would you need? What practical obstacles would you face?

C.2. Propose a mechanism for your field that would make the "file drawer" visible — that would ensure negative results, failures, and null findings enter the evidence base alongside positive results.

C.3. The chapter notes that registered reports produce different findings than traditional publications. Design an experiment to quantify the difference in your field.

Part D: Synthesis & Critical Thinking ⭐⭐⭐

D.1. The chapter argues that survivorship bias and publication bias interact with the authority cascade (Chapter 2) and the streetlight effect (Chapter 4). Trace these interactions for a specific case.

D.2. Is the chapter's own argument subject to survivorship bias? (Does it select historical examples that support its thesis while ignoring cases where survivorship bias didn't produce problems?) Evaluate this critique.

D.3. The "calibrated confidence" approach recommends holding conclusions with less certainty than surviving evidence suggests. But how much less? Is there a principled way to determine the discount, or is it inevitably subjective?

D.4. Compare Wald's approach (looking for what's missing) with Popper's approach from Chapter 3 (looking for what would disprove). Are they different expressions of the same principle?

Part M: Mixed Practice (Interleaved) ⭐⭐–⭐⭐⭐

M.1. (From Chapter 4) How does the streetlight effect create survivorship bias? If fields study what's measurable, does the evidence from unmeasurable dimensions get "filed away" in a metaphorical drawer?

M.2. (From Chapter 3) Apply the falsifiability diagnostic to the claim "successful companies share common traits." Is this claim falsifiable? How does survivorship bias in the evidence base affect its falsifiability?

M.3. (From Chapter 2) The dietary fat hypothesis was maintained partly by an authority cascade (Ancel Keys) and partly by publication bias (studies contradicting the fat hypothesis were harder to publish). Trace both mechanisms and explain how they reinforced each other.

M.4. (Integration) Update your Epistemic Audit with the survivorship bias lens. What evidence in your field's file drawer might change the current consensus?

Part E: Research & Extension ⭐⭐⭐⭐

E.1. Read the Turner et al. (2008) antidepressant publication bias study (or a summary of it). Write a 1,000-word analysis applying the chapter's framework.

E.2. Investigate the replication rates in your field (if available). How do they compare to the 36% replication rate found in psychology? What role might publication bias play?

Solutions

Selected solutions in appendices/answers-to-selected.md.