Quiz: Survivorship Bias at Scale

Target: 70% or higher to proceed confidently.

Section 1: Multiple Choice (1 point each)

1. Abraham Wald recommended armoring bomber engines because: - A) Engines were the most expensive component to replace - B) The returning bombers showed engine damage, indicating engines were vulnerable - C) The returning bombers showed few engine hits, indicating that engine hits were fatal (those planes didn't return) - D) Engine armor was lighter than fuselage armor

Answer**C)** The absence of engine hits on surviving planes meant engine hits were fatal — those planes crashed. The surviving evidence showed where planes could be hit and survive, not where they were most vulnerable. *Reference:* Section 5.1

2. Publication bias is a form of survivorship bias because: - A) Only the best-written papers survive peer review - B) Published studies (positive results) survive to enter the evidence base while unpublished studies (null results) are filtered out - C) Old publications are replaced by newer ones - D) Only well-funded studies get published

Answer**B)** The publication process acts as a selection filter that systematically preserves positive results and removes null/negative results. *Reference:* Section 5.4

3. In the antidepressant publication bias study, published literature showed what percentage of studies as positive? - A) 51% - B) 73% - C) 94% - D) 100%

Answer**C)** 94% appeared positive in published literature vs. ~51% in the full FDA-registered data. *Reference:* Section 5.4

4. The "denominator problem" in business success literature means: - A) Companies don't have enough employees - B) Studies examine only successful companies without comparing them to failed companies with similar characteristics - C) Financial ratios are calculated incorrectly - D) Success rates are measured in the wrong units

Answer**B)** Without the denominator (failed companies with similar characteristics), you can't distinguish causal factors from coincidental ones. *Reference:* Section 5.2

5. The "halo effect" in business research refers to: - A) Successful companies having good public relations - B) Retroactively relabeling the same characteristics as positive or negative depending on the company's outcome - C) Executives appearing more attractive when their companies succeed - D) Companies benefiting from association with other successful companies

Answer**B)** When a company succeeds, its traits are labeled positively ("bold strategy"); when it fails, the same traits are relabeled negatively ("reckless strategy"). *Reference:* Section 5.2

6. A funnel plot detects publication bias by showing: - A) Which journals publish the most studies - B) Asymmetry in the distribution of study results — missing small negative studies - C) The chronological trend of publications - D) Which researchers are most productive

Answer**B)** An asymmetric funnel plot — with small negative studies missing — is the visual signature of publication bias. *Reference:* Section 5.4

Section 2: True/False with Justification (1 point each)

7. "Survivorship bias makes things look worse than they are."

Answer**False.** Survivorship bias always makes things look *better* than they are, because the failures are removed from the evidence. Published studies overstate treatment effects. Business literature overstates the replicability of success strategies. Fund performance data overstates expected returns.

8. "Pre-registration of studies completely eliminates publication bias."

Answer**False.** Pre-registration reduces publication bias by committing to an analysis plan in advance, but it doesn't eliminate it entirely. Researchers can still conduct unregistered studies, selectively report pre-registered outcomes, or abandon pre-registered studies that produce unwanted results. Registered reports (where journals commit to publish regardless of results) are a stronger mechanism but still not universal.

9. "The business success literature is worthless because of survivorship bias."

Answer**False (overstated).** Survivorship bias makes the literature unreliable as a *causal* account of what drives success, but the descriptions of what successful companies *did* can still be informative as case studies — provided readers understand that the same approaches failed at many other companies. The problem is treating survivor-biased observations as causal recipes rather than descriptive illustrations.

Section 3: Short Answer (2 points each)

10. Apply Wald's five-step diagnostic template to the following: "Social media influencers say the key to success is posting consistently." Identify each step.

Sample Answer 1. Evidence: Successful influencers report that consistency was key to their growth 2. Selection filter: Only successful influencers are visible and interviewed; unsuccessful ones are invisible 3. What was excluded: The millions of creators who posted consistently and still failed 4. Does the filter correlate? Yes — success is the filter, and success is the variable of interest. The filter removes exactly the counter-evidence we need. 5. Adjustment: Consistency is probably necessary but not sufficient. The surviving evidence cannot tell us whether it's a differentiating factor.

Section 4: Applied Scenario (3 points)

11. A pharmaceutical company presents data from three clinical trials of a new drug, all showing positive results (p < 0.05). A physician is deciding whether to prescribe the drug. Using the chapter's framework, what questions should the physician ask before deciding?

Sample Answer Key questions: (1) How many total trials were conducted? If five trials were done and only three were published, publication bias is likely. (2) Were these trials pre-registered? If yes, the results are more trustworthy. (3) Is there a funnel plot analysis available? Asymmetry would suggest missing negative studies. (4) Were any of these trials funded by the company? Industry funding correlates with higher publication bias. (5) Are there systematic reviews that include unpublished data (e.g., from FDA registration databases)? (6) What is the base rate for this class of drugs? How many similar drugs have been tested and failed? The physician should calibrate confidence below what three positive published studies would normally warrant, because publication bias means the full evidence base is likely less favorable than the published subset.

Scoring & Next Steps

Score Assessment Recommended Action
< 50% Needs review Re-read 5.1–5.2 and 5.4
50–70% Partial Review Wald's template and the antidepressant case
70–85% Solid Ready to proceed
> 85% Strong Proceed to Chapter 6