Appendix D: Research Methods Primer

How to evaluate the research this book references — and the claims that will arrive in your feed.

Why This Appendix Exists

This book cites research. So do creators who discuss psychology, social media, behavior, and health in their content. So does every article you'll share or discuss in your videos. Being able to evaluate the quality of that research — not just whether a study exists, but whether it means what the headline says it means — is one of the most important skills a creator can develop.

This primer won't make you a research scientist. It will make you a more careful reader of research claims, which is sufficient for most creator purposes.

The Research Quality Spectrum

Not all research is equally reliable. From most to least reliable:

Meta-analyses and systematic reviews Combine and analyze results from many individual studies. Because they aggregate evidence, they're more reliable than any individual study. When you see "meta-analyses show that X," this is the strongest form of evidence. Still not infallible — depends on the quality of included studies and the review methodology.

Randomized controlled trials (RCTs) The "gold standard" for studying cause-and-effect. Participants are randomly assigned to conditions; researchers manipulate one variable and measure outcomes. Because of randomization, we can be more confident that differences are caused by the manipulation, not by pre-existing differences between groups. Less common in psychology/media research than in medicine.

Longitudinal studies Follow the same participants over time. Better than single-point studies for understanding how things change and what predicts what. Still observational — cannot establish causation as reliably as an RCT.

Cross-sectional studies Measure a group of people at one point in time. Most common type in social media research. Can establish associations but not causation. When you see "social media use is correlated with depression," this is usually a cross-sectional finding.

Case studies and qualitative research In-depth examination of one or a few cases. Valuable for generating hypotheses and understanding mechanisms; not designed to establish prevalence or causation.

Anecdote and expert opinion Not research. One person's experience, even a very smart person's experience, is not evidence of a general pattern. Valuable as illustration; not as evidence.

The Correlation-Causation Problem

The most common misrepresentation of research in media and creator content: treating correlation as causation.

Correlation: Two things are related — when one goes up, the other tends to go up (or down).

Causation: One thing causes the other — changing the first thing produces a change in the second.

The problem: Correlation does not establish causation. There are always alternative explanations:

Example: "Social media use is associated with higher rates of depression in teenagers."

This could mean: - Social media use causes depression (causation as stated) - Depression causes social media use (reverse causation — depressed teens turn to social media) - Something else causes both (a third variable — loneliness causes both more social media use AND depression) - All three operate simultaneously

Cross-sectional studies cannot distinguish between these. Longitudinal studies can partially distinguish them. RCTs can establish causation. Most social media + mental health research is cross-sectional.

The creator application: When you see a study, ask: Is this just showing two things are related, or does the study design actually establish that one causes the other? If you're not sure, hedge your language accordingly.

Effect Sizes: Does It Actually Matter?

A study can find a statistically significant effect that is so small it has no practical importance.

Statistical significance means: this difference is unlikely to have occurred by chance (typically defined as less than 5% probability). Does NOT mean the difference is large or important.

Effect size measures how large the difference actually is. Common measures: - Cohen's d: small = 0.2, medium = 0.5, large = 0.8 - r (correlation): small = 0.1, medium = 0.3, large = 0.5

The Orben-Przybylski example (Chapter 38): These researchers reanalyzed large social media datasets and found that the effect sizes for technology use on adolescent wellbeing were comparable in magnitude to the effect of wearing glasses or eating potatoes. These were statistically significant — but so small as to be practically meaningless for most individuals. Contrast with the large Twenge studies that generated headlines like "smartphones are destroying a generation."

Both findings can be true simultaneously: the effect is real and statistically significant; AND the effect is so small that the headline framing substantially overstates its importance.

When evaluating research claims, ask: How big is the effect? Does the researchers' own paper describe it as large, medium, or small?

Replication: The Scientific Check

A single study finding is a preliminary observation, not established fact. Science builds confidence through replication — other researchers, in other labs, with other samples, finding the same result.

High replication reliability: Physics, chemistry, many biology findings. The laws of motion replicate perfectly; so do most cell biology findings.

Variable replication reliability: Psychology and social science have experienced a "replication crisis" — many well-known findings have failed to replicate or replicated with smaller effects. Famous examples: the ego depletion effect (willpower runs out like a resource), many priming effects, power poses.

This doesn't mean social science is useless. It means: single studies, even prestigious ones in top journals, are not definitive. Findings that have replicated across many studies and populations are more reliable than findings from a single famous study.

Creator application: When a finding has been replicated many times across different researchers and populations, you can state it with reasonable confidence. When it comes from a single study, hedge: "one study found..." or "preliminary research suggests..."

Reading a Research Abstract

Most journal articles are behind paywalls, but abstracts (summaries) are usually free. When you have access to an abstract:

Title: States the research question or finding Authors and institution: Who conducted this? Methods: How was the study designed? (Survey? Experiment? How many participants?) Results: What did they find? (Look for effect sizes, not just "significant") Conclusion: What do the authors say it means?

Critical questions: 1. How many participants? (Larger samples = more reliable; under 100 is small for quantitative research) 2. Who were the participants? (College students ≠ general population; US sample ≠ universal) 3. Does the conclusion match the data? (Authors sometimes overclaim in conclusions) 4. What alternative explanations does the study not rule out?

Primary Sources vs. Summaries

Most research reaches creators through: - News articles summarizing a study - Social media posts referencing a study - Other creators discussing research - Books citing studies

Each level of translation introduces potential for misrepresentation. The classic telephone game.

The creator's accuracy standard (from Chapter 38 in action): For any specific factual claim, trace it to the primary source. What did the actual study find? How large was the sample? What were the limitations the authors themselves acknowledged?

"Studies show that social media causes depression" is not a claim that can be sourced to any study, because no study shows this. "Cross-sectional studies have found correlations between social media use and self-reported depression symptoms, particularly among adolescent girls, though causation has not been established" is accurate to the research landscape.

The first is easier to say. The second is what the evidence actually supports.

A Practical Checklist for Evaluating Research Claims

Before sharing or citing any research-based claim:

□ Can I find the original study? (Not just the article that summarizes it)

□ What type of study is it? (RCT, longitudinal, cross-sectional, case study — and what does that mean for the strength of the conclusion?)

□ What was the sample? (Size, population, context — and does it represent who the claim is being made about?)

□ What does the effect size say? (Is the effect large, medium, or small?)

□ Has it been replicated? (Is this one study, or a pattern across many?)

□ Does the summary match the study? (Read what the authors actually conclude in the paper, not just the press release)

□ What alternative explanations weren't ruled out? (What else could explain the finding?)

□ Am I hedging appropriately? ("Research suggests" rather than "research proves"; "associated with" rather than "causes")

This primer won't prevent you from ever getting something wrong — everyone does. What it will do is reduce the probability of getting it wrong in ways that matter: confidently stating something as fact that the evidence only tentatively supports, or sharing a finding as established when it's actually contested.

The standard isn't perfection. It's good-faith engagement with what the evidence actually says, clearly labeled with the confidence that evidence actually warrants.