Case Study 2: Publication Bias in Antidepressant Research

The Discovery

In 2008, Ewan Turner and colleagues published a study in the New England Journal of Medicine that would reshape understanding of how publication bias distorts medical evidence. Their method was simple but powerful: compare what the FDA knew (from mandatory trial registration) with what physicians knew (from the published literature).

The FDA requires pharmaceutical companies to register all clinical trials and submit all results — positive, negative, and ambiguous. The published literature, by contrast, contains only what journals choose to publish and researchers choose to submit. By comparing these two bodies of evidence, Turner could measure the exact size and direction of the publication bias.

The Findings

The study examined 74 FDA-registered trials of 12 antidepressant drugs approved between 1987 and 2004.

The FDA Picture

  • 38 trials had positive results (the drug worked)
  • 36 trials had negative or questionable results (the drug didn't clearly work)
  • Overall assessment: antidepressants modestly effective, with substantial variability

The Published Literature Picture

  • 37 of 38 positive trials were published (97%)
  • Only 3 of 36 negative trials were published as negative (8%)
  • 11 negative trials were published but with results reframed to appear positive
  • 22 negative trials were never published at all
  • Overall appearance: antidepressants highly effective, with consistent evidence

The Distortion

The published literature made antidepressants appear to have an average effect size of 0.37 (a moderate effect). The full FDA database showed an average effect size of 0.15 (a small effect) — less than half what the published literature suggested.

The Consequences

This distortion had measurable consequences for clinical practice:

  • Prescribing decisions were based on an inflated evidence base. Physicians who relied on the published literature believed antidepressants were more effective than they actually were.
  • Patient expectations were calibrated to the inflated evidence. Patients who didn't respond to antidepressants may have felt that something was wrong with them, when in fact the drugs simply weren't as effective as the literature claimed.
  • Treatment alternatives were undervalued. If antidepressants appear highly effective, there's less motivation to invest in psychotherapy, exercise, social support, or other interventions that might be comparably effective.
  • Cost-benefit calculations were distorted. The cost of antidepressant prescriptions (including side effects, long-term dependence, and withdrawal symptoms) was weighed against an inflated estimate of benefits.

Structural Analysis

This case perfectly illustrates every element of the chapter's framework:

Element How It Operated
Selection filter Publication process selected positive studies for inclusion
Missing evidence 22 negative studies never published; 11 more reframed
Correlation with variable of interest The filter (publication) directly correlated with the variable (drug efficacy)
Direction of bias Always toward overestimation of drug effectiveness
Structural causation No individual fraud needed — the incentive system produced the bias emergently

The Response

The antidepressant publication bias study contributed to several reforms:

  • Trial registration requirements were strengthened, making it harder to conduct unreported trials
  • The AllTrials campaign advocated for all clinical trial results to be made public
  • Journals began adopting policies requiring trial registration as a condition of publication
  • Systematic reviewers became more aware of the need to search for unpublished evidence

However, the reforms remain incomplete. Many trials are still registered but not reported. "Outcome switching" (changing which outcomes are reported after seeing the data) continues. And the fundamental incentive structure — journals preferring positive results, companies preferring positive publicity — remains largely unchanged.

Discussion Questions

  1. If you were a physician who learned about this study in 2008, how would it change your prescribing practice? Be specific.
  2. The publication bias wasn't caused by fraud — it was an emergent property of the incentive system. Does this make it more or less concerning than deliberate fraud? Why?
  3. Design a drug evaluation system that would be immune to publication bias. What trade-offs would your system involve?
  4. Apply the antidepressant analysis to another drug class or medical intervention. What would you expect to find?

References

  • Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy." New England Journal of Medicine, 358(3), 252–260. (Tier 1)
  • Goldacre, B. (2012). Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients. Fourth Estate. (Tier 1)
  • The AllTrials campaign (alltrials.net) documents ongoing efforts to ensure all clinical trial results are reported. (Tier 2)
  • Rosenthal, R. (1979). "The File Drawer Problem and Tolerance for Null Results." Psychological Bulletin, 86(3), 638–641. (Tier 1 — the original formulation)