Chapter 4: Key Takeaways

The 9-Step Fact-Checker's Toolkit

Step Question Why It Matters
1 What is the specific claim? Vague claims can't be evaluated. Pin it down.
2 What is the original source? No source = folk wisdom, not science.
3 Single study or meta-analysis? Meta-analyses > replications > single studies.
4 What was the sample? Size, demographics, WEIRD problem.
5 Has it been replicated? The single most important question post-replication crisis.
6 What is the effect size? Significant ≠ large. Most psych effects are small to medium.
7 What do other experts say? One researcher's claim ≠ scientific consensus.
8 Who benefits? Incentives don't mean falsehood, but they warrant scrutiny.
9 Too good to be true? Simple, dramatic, universal claims → be suspicious.

Core Concepts

  1. The toolkit is a permanent skill. Once learned, it applies to every psychology claim you encounter for the rest of your life — on social media, in books, at work, from friends, and in this book.

  2. Statistical significance ≠ practical importance. A p-value below .05 means the result is unlikely due to chance. It does not tell you the effect is large, important, or worth caring about. Always check the effect size.

  3. The hierarchy of evidence matters. Meta-analyses with publication bias corrections > large pre-registered replications > multiple independent replications > single large study > single small study > no empirical evidence.

  4. Unfalsifiable claims are not scientific claims. If a claim can't be stated specifically enough to be tested and potentially proven wrong, it is not a scientific claim, regardless of how it's labeled.

  5. "I don't know yet" is a valid and important conclusion. Not every claim resolves neatly. Some evidence is genuinely mixed, and the honest answer is uncertainty. The ability to tolerate uncertainty is a critical thinking skill.

  6. Apply the toolkit to this book too. No source is exempt from scrutiny, including this one.

Evidence Ratings in This Chapter

Claim Rating Summary
"People are left-brained or right-brained" ❌ DEBUNKED No fMRI evidence for hemispheric dominance in personality
"Your brain replaces itself every 7 years" ❌ DEBUNKED Most neurons are never replaced; radiocarbon dating confirms
"You can learn to evaluate psychology claims with a framework" ✅ SUPPORTED Critical thinking instruction produces moderate improvements
"Couples who share housework equally are 50% less likely to divorce" ⚠️ OVERSIMPLIFIED Real but modest correlation; specific percentage unsupported

Key Terms Introduced

  • Effect size (Cohen's d, correlation r): The magnitude of a finding, distinct from its statistical significance
  • Hierarchy of evidence: The ranking of evidence types from most reliable (meta-analyses) to least (anecdotes)
  • Unfalsifiable claim: A claim that cannot, in principle, be proven wrong — and therefore is not a scientific claim
  • TGTBT test: The "too good to be true" test for claims that are suspiciously clean, dramatic, or universal

One Sentence to Remember

The 9-step toolkit doesn't tell you what to believe — it tells you what questions to ask, and those questions are almost always more valuable than the answers.