Appendix A: Research Methods in Propaganda Studies

"The tools we use to study persuasion shape what we can see. No single method illuminates the whole phenomenon. Good researchers use multiple lenses."

This appendix serves two purposes. First, it helps you interpret the research cited throughout this textbook — understanding how a study was designed tells you what it can and cannot prove. Second, it guides you through the methodologies available if you conduct original research, whether for the Inoculation Campaign community analysis, a course paper, or independent inquiry.

Propaganda studies draws from political science, social psychology, communication studies, history, and computer science. Each discipline brings its own methodological traditions, strengths, and blind spots. Becoming a careful reader of research in this field means learning to evaluate claims across all of them.

Section 1: Overview of Research Approaches

Qualitative, Quantitative, and Mixed-Methods Research

Research in propaganda and media influence generally falls into three broad methodological families.

Quantitative research uses numerical measurement and statistical analysis. It asks questions like: How many times does a given propaganda technique appear in a corpus of political ads? Do subjects who receive inoculation messages show measurably lower susceptibility to misinformation? What percentage of accounts spreading a specific hashtag are automated bots? Quantitative research is strongest when you need to establish patterns across large datasets, test causal hypotheses with controlled experiments, or track changes over time with precision.

Qualitative research uses non-numerical interpretation — close reading, interview analysis, ethnographic observation. It asks questions like: How do propaganda techniques construct a particular vision of national identity? How do individuals who participated in a disinformation campaign describe their motivations? What does it feel like, from inside an online community, to be subjected to coordinated narrative manipulation? Qualitative research is strongest when you need to understand meaning, context, and lived experience rather than frequency and effect size.

Mixed-methods research combines both. A study might use computational analysis to identify which accounts spread a disinformation narrative at scale, then conduct in-depth interviews with people exposed to that narrative to understand how (if at all) it shaped their beliefs. Mixed methods are increasingly standard in propaganda research because the phenomenon is simultaneously a mass-media event, a psychological experience, and a historically situated social practice.

Why No Single Method Captures the Full Phenomenon

Propaganda works at multiple levels simultaneously, and each level requires different tools. A piece of political advertising has a formal structure (amenable to content analysis), a psychological effect on viewers (amenable to experimental study), a production history involving specific institutions and decisions (amenable to archival research), and a social life in which it circulates and is interpreted (amenable to ethnography and network analysis). Research that captures only one level risks missing the causal story entirely.

This is not a weakness unique to propaganda studies. It reflects the genuine complexity of any social phenomenon that involves mass communication, individual psychology, institutional power, and historical context at once.

The Levels-of-Analysis Problem

Researchers disagree — often productively — about which level of analysis is most important for explaining propaganda's effects.

Individual level: How does exposure to a specific message change what a particular person believes or how they behave? This is the domain of experimental psychology and survey research.
Group level: How do communities form around shared narratives, and how do in-group dynamics amplify or dampen propaganda effects? This is the domain of social psychology, network analysis, and group-level ethnography.
Institutional level: Who produces propaganda, under what organizational constraints, with what budgets and strategic intentions? This is the domain of political economy, organizational sociology, and investigative journalism.
Societal level: How does the cumulative, long-term operation of propaganda reshape the epistemic environment — the shared pool of facts and values a society draws on? This is the domain of historical analysis, critical theory, and longitudinal survey research.

Each level of analysis calls for different methods. When you encounter a study, ask: what level is this operating at, and what does that mean for the claims it can legitimately make?

Section 2: Content Analysis

Definition and Purpose

Content analysis is the systematic, replicable examination of communication content to identify patterns, frequencies, and relationships. In propaganda research, it is used to answer questions like: What rhetorical techniques appear most frequently in state television broadcasts? How has the framing of immigration changed across a decade of newspaper front pages? Which emotional appeals dominate a set of political Facebook ads?

The key word is systematic. Unlike close reading (which is interpretive and focused on individual texts), content analysis creates a structured coding scheme, applies it consistently across a large corpus, and produces quantitative output that can be analyzed statistically.

Manifest vs. Latent Content

Manifest content is the surface-level, explicit content of a text — the words that appear, the images shown, the claims stated outright. Coding manifest content is relatively straightforward because it requires minimal interpretation: does this ad use the word "crisis"? Does this broadcast show images of violence?

Latent content involves the underlying meaning, tone, or implication — the message beneath the surface. Coding latent content requires trained judgment: does this ad imply that immigrants are criminals without stating it explicitly? Is this headline designed to provoke fear? Latent content coding is more powerful but harder to do reliably, because coders must apply consistent interpretive frameworks to texts that are intentionally ambiguous.

Unit of Analysis

Before coding, researchers must define the unit of analysis — the basic element being counted. Common choices include:

Word or phrase (useful for identifying specific terms or themes)
Sentence or claim (useful for fact-checking-style analysis)
Article, broadcast segment, or post (useful for comparing sources or time periods)
Campaign or message series (useful for analyzing coordinated influence operations)

The unit of analysis shapes what you can discover. Coding at the sentence level reveals the distribution of claims within a text; coding at the article level reveals the distribution of coverage across an outlet.

Inter-Rater Reliability

Because content coding involves human judgment, researchers must demonstrate that their coding is not idiosyncratic — that other trained researchers applying the same scheme would arrive at the same results. This is measured through inter-rater reliability: two or more coders independently code a sample of the corpus, and their agreement is quantified.

The most commonly reported statistic is Cohen's kappa, which measures agreement adjusted for chance. A kappa above 0.70 is generally considered acceptable; above 0.80 is good. If two coders coding "propaganda techniques" agree only at the level expected by chance, the coding scheme needs revision — either the categories are unclear, or the latent content is genuinely ambiguous.

Strengths and Limitations

Content analysis is systematic, replicable, and scalable to large volumes of text (especially when combined with computational methods). It is the appropriate tool for answering "what is in this corpus?" questions.

Its fundamental limitation is that it cannot answer "what effect does this content have?" questions. A study showing that 73% of political ads in a given election cycle used fear appeals tells you something important about the information environment — but it tells you nothing about whether those appeals actually changed voters' beliefs or behavior. Effect questions require experimental or longitudinal survey designs.

Example Application

Researchers studying disinformation on social media often use content analysis to categorize posts by technique (false context, fabricated content, misleading framing), by topic (health, elections, immigration), and by emotional valence (fear, anger, disgust, hope). A typical study of this kind might involve three trained coders applying a scheme to 500 posts, reporting a kappa of 0.78, and then analyzing patterns computationally across a larger uncoded corpus using the validated scheme as a training signal for machine learning.

Section 3: Experimental Methods

Why Experiments Are Powerful

Experiments are the only research design that can establish causation rather than mere correlation. This matters enormously in propaganda research, where the central question is often not just "do people who consume state media hold different beliefs?" but "does state media cause different beliefs?"

The logic of the experiment is simple: if you randomly assign participants to receive or not receive a treatment (a propaganda message, an inoculation intervention, a fact-check), and you observe different outcomes between the groups, then the treatment caused the difference — because random assignment makes the groups equivalent on everything else.

Lab Experiments in Propaganda Research

Lab experiments expose participants to controlled stimuli in a controlled environment and measure outcomes immediately after. In inoculation research (discussed extensively in Chapter 33), the typical design involves:

Pre-measurement: Assess participants' baseline beliefs on a topic, and their susceptibility to the propaganda technique being studied.
Random assignment: Participants are randomly assigned to the inoculation condition (they receive a warning and a brief refutation of the manipulation technique) or a control condition (they receive neutral content).
Exposure: All participants are then exposed to the actual propaganda or misinformation message.
Post-measurement: Assess beliefs, attitude change, and ability to identify the manipulation technique.

The dependent variable (DV) — what you measure to assess the effect — might be belief in the misinformation claim, attitude change toward the topic, or ability to correctly identify the propaganda technique used.

Survey Experiments

Survey experiments embed experimental stimuli directly in an online survey. Participants are randomly shown different versions of a question, article, or message. This design allows researchers to test effects in a more naturalistic context (people take surveys at home on their own devices) while maintaining the causal power of random assignment.

Survey experiments are commonly used to test the effects of fact-checks, source labels, and message framing on belief and sharing behavior.

Field Experiments

Field experiments test interventions in real-world settings with real stakes. Researchers partner with platforms, governments, or NGOs to randomly assign treatments to users, communities, or regions. Field experiments have high external validity — their findings generalize to the real world because they are conducted in the real world.

The challenge is feasibility and ethics. Randomly exposing some communities to propaganda or withholding fact-checks from others raises serious ethical concerns. The most common field experiments in this domain test positive interventions — media literacy curricula, inoculation campaigns, or accuracy nudges — rather than manipulating harmful content exposure.

Limitations: Internal vs. External Validity

The fundamental trade-off in experimental research is between internal validity (confidence that the effect was caused by the treatment) and external validity (confidence that the effect generalizes beyond the study setting). Lab experiments maximize internal validity at the cost of realism. Field experiments maximize realism at the cost of control.

The "lab-to-field gap" — the phenomenon in which laboratory findings fail to replicate in real-world settings — is a central methodological concern in propaganda research (see Chapter 33). Inoculation effects demonstrated in the lab may be smaller, more condition-dependent, or shorter-lived in the wild.

Example: How Inoculation Studies Are Designed

Sander van der Linden and colleagues' foundational inoculation studies randomly assign participants to receive either a forewarning ("You are about to see information that uses the technique of fake experts — here is how that technique works") or a control. Participants then see actual climate misinformation using that technique. Post-exposure, inoculated participants show less attitude change and better technique recognition. The design controls for prior beliefs, media consumption habits, and political identity through randomization across conditions of hundreds of participants.

Section 4: Survey Research

Cross-Sectional vs. Longitudinal Surveys

A cross-sectional survey captures a snapshot at a single point in time: it tells you the distribution of beliefs, behaviors, and media habits in a population at the moment of measurement. Cross-sectional surveys are relatively cheap and fast but cannot track change or establish causation.

A longitudinal survey (also called a panel survey) follows the same respondents over multiple time points. Panel surveys are the workhorse of propaganda effects research because they allow researchers to ask whether changes in media exposure precede changes in beliefs — a necessary (though not sufficient) condition for causal inference.

The Media Consumption and Beliefs Survey

The standard survey instrument in propaganda and media effects research typically combines:

Demographic measures: age, education, political identity, income
Media consumption measures: source use, frequency, trust in different outlet types
Belief and attitude measures: specific factual beliefs (often on contested empirical questions), attitudinal positions, confidence levels
Epistemic measures: trust in institutions, perception of media bias, belief in conspiracy theories

By analyzing the relationships among these variables across the population, researchers can describe who consumes what media and what they believe — a necessary precursor to causal research.

A significant challenge in survey research on propaganda and disinformation is social desirability bias: respondents may not accurately report beliefs or behaviors they perceive as stigmatized. Someone who believes a fringe conspiracy theory may not admit it to a survey researcher. Someone who shares misinformation may underreport that behavior.

Researchers address this with list experiments (in which respondents indicate how many of a list of statements they agree with, but not which ones), implicit attitude measures, and behavioral measures (tracking actual sharing behavior through consented data access rather than self-report).

Example: Reuters Institute Digital News Report

The Reuters Institute Digital News Report is one of the most widely cited annual surveys of media consumption and trust globally. It surveys nationally representative samples in 40+ countries, asking about news consumption habits, source trust, and attitudes toward information quality. Its methodology — online panels with demographic quota sampling — represents a common standard in the field, and its limitations (non-probability sampling, English-language dominance, survey fatigue) are regularly discussed by its authors.

Section 5: Network Analysis and Computational Methods

Social network analysis (SNA) treats the spread of information as a network phenomenon: accounts are nodes, and interactions (shares, replies, retweets) are edges. Analyzing the topology of this network reveals things no content analysis can: Who are the high-centrality nodes that connect otherwise separate communities? Does a disinformation narrative flow from fringe accounts to mainstream accounts, or vice versa? Are there clusters of accounts that consistently amplify each other's content?

SNA applied to disinformation spread can reveal coordinated behavior that individual post analysis would miss: a thousand accounts that each post once look very different from a thousand accounts that systematically share each other's content within seconds of posting.

Computational Propaganda Research

Computational propaganda research uses algorithmic methods to analyze large-scale online behavior. Key methods include:

Bot detection: Using behavioral signals (posting frequency, account age, linguistic patterns) to identify likely automated accounts. The Oxford Internet Institute's Computational Propaganda Project has documented bot activity in elections across dozens of countries using these methods.
Coordinated inauthentic behavior identification: Looking for accounts that post the same content at the same time, suggesting coordination rather than organic behavior.
Topic modeling: Using statistical methods (Latent Dirichlet Allocation, or more recently transformer-based models) to identify themes across large corpora of posts.

Stanford Internet Observatory / Graphika Methodology

Organizations like the Stanford Internet Observatory and Graphika have developed a standard methodology for documenting influence operations: collect a dataset of accounts and content associated with a suspected operation, analyze network topology and content patterns, identify behavioral fingerprints (coordinated posting, template reuse, account provenance), and cross-reference with known infrastructure (domain registrations, ad spending records, leaked documents). Their reports serve as methodological models for accountability journalism and platform-facing research.

Natural Language Processing in Propaganda Analysis

Large language models (LLMs) and NLP tools increasingly enable researchers to analyze text at a scale previously impossible. Techniques include:

Sentiment analysis (detecting emotional valence)
Named entity recognition (identifying people, places, and organizations mentioned)
Stance detection (identifying whether a text supports, opposes, or is neutral toward a claim)
Narrative framing analysis (identifying recurring story structures)

These tools accelerate content analysis but introduce their own validity questions: models trained on general text may not perform well on political propaganda, non-English content, or deliberate obfuscation.

Limitations

The fundamental limitation of computational propaganda research is data access. Platform APIs (the programmatic interfaces researchers use to collect data) have been progressively restricted since 2018. The Twitter/X API changes of 2023 effectively ended large-scale academic Twitter research for most researchers without substantial institutional funding. Researchers now operate with limited, often non-representative data samples. Results should be interpreted in light of what could not be observed.

Section 6: Archival and Historical Methods

The Primary Source Archive

Historical propaganda analysis rests on the primary source archive: the original documents, broadcasts, internal communications, and material artifacts produced by propaganda campaigns. Primary sources let researchers answer questions about intent, organization, and process that no survey or experiment can reach.

A historian studying the Creel Committee's domestic propaganda during World War I has access to committee records, pamphlets, poster production records, and correspondence. This documentary trail reveals decisions made, alternatives rejected, and institutional logics that shaped the campaign's form. No contemporary survey of American attitudes could reconstruct this.

Key Propaganda Document Archives

Several archives are essential resources for propaganda researchers:

UCSF Truth Tobacco Industry Documents Library: 14 million internal documents from tobacco company litigation discovery, revealing decades of coordinated disinformation about smoking health risks. This archive is the model for how litigation-produced disclosure can make private propaganda infrastructure publicly legible.
National Archives and Records Administration (NARA): U.S. government records including declassified propaganda program materials, policy documents, and strategic communications records.
Library of Congress: Extensive collections of historical American propaganda materials, wartime posters, government pamphlets, and congressional records.
Harvard's Baker Library Historical Collections: Business history collections including public relations industry records.

Declassified Intelligence Documents

FOIA (Freedom of Information Act) requests can surface documents that were classified at the time of their production. Researchers studying government propaganda programs — including domestic ones like COINTELPRO or international ones like USIA operations — rely on declassified documents to reconstruct programs that were never intended to be publicly known. The NSA, CIA, and State Department have all produced substantial declassified archives on influence operations through FOIA litigation.

Industry Document Litigation Discovery

Litigation against tobacco companies, pharmaceutical manufacturers, and fossil fuel companies has produced massive troves of internal documents revealing coordinated disinformation strategies. These documents — produced under legal compulsion — are uniquely valuable because they are unmediated by institutional public relations: they show what operatives actually said to each other, not what they said publicly. Naomi Oreskes and Erik Conway's research for Merchants of Doubt relied heavily on this method.

Close Reading as Method

"Close reading" — sustained, careful attention to the specific language, structure, and rhetorical choices of a text — is itself a methodological practice, not just an informal habit. When historians or rhetorical scholars analyze a propaganda document, they are applying trained interpretive skills: attending to word choice, identifying appeals and their targets, reconstructing the implied audience, situating the text in its institutional and historical context. Close reading produces interpretive claims that must be defended with evidence from the text and its context — it is rigorous, even if its rigor looks different from statistical analysis.

Section 7: Ethnographic and Interview Methods

Why Qualitative Methods Matter

Quantitative and computational methods can tell you that disinformation spreads faster than corrections, that certain demographic groups consume more state-aligned media, or that bot accounts amplify specific narratives. They cannot tell you what it feels like to be inside a community where conspiracy theories circulate as shared knowledge, or how a propaganda practitioner rationalizes their work. For these questions, qualitative methods — interviews, ethnography, focus groups — are essential.

In-Depth Interviews

In-depth interviews with propaganda practitioners, disinformation researchers, platform employees, and affected community members produce detailed, contextualized accounts that surveys cannot. They are appropriate when the research question concerns subjective experience, institutional process, or decision-making logic.

Effective interviewing requires careful question design (avoiding leading questions, building from general to specific), attention to rapport and power dynamics, and rigorous analysis of transcripts. Interview data is typically analyzed through thematic coding — identifying recurring themes and patterns across multiple accounts.

Ethnographic Research in Online Communities

Ethnography — sustained, immersive observation of a community in its own context — has been adapted for online settings. Researchers studying how disinformation circulates in specific communities (QAnon forums, anti-vaccine Facebook groups, nationalist Telegram channels) must navigate significant methodological and ethical challenges:

Access: Some communities are closed or hostile to researchers. Covert access raises serious ethical problems.
Consent: Standard IRB requirements for informed consent are difficult to implement in public-but-sensitive online spaces.
Interpretation: Researchers must resist projecting their own frameworks onto communities whose internal logic may be radically different from their assumptions.
Amplification: Describing an extremist community in detail may inadvertently publicize its existence and draw new members.

Kate Starbird's research on crisis misinformation ecosystems and Alice Marwick and Rebecca Lewis's work on the alt-right media ecosystem are models of careful ethnographic and network research in these difficult contexts.

Focus Groups

Focus groups bring small groups of participants together to discuss a topic in a moderated setting. They are useful for exploring how communities interpret and respond to propaganda or counter-messaging campaigns — not to produce statistically generalizable findings, but to illuminate the range of responses and the reasoning behind them. Media literacy researchers often use focus groups to pre-test educational interventions before field deployment.

Defector Testimony

In research on authoritarian propaganda systems, former propagandists, state media employees, and military information operations personnel who have left their positions offer uniquely valuable accounts of how these systems work from the inside. "Defector testimony" — systematic interview research with individuals who have left propaganda-producing institutions — can reveal organizational structure, decision-making processes, and strategic intent that no amount of content analysis can reconstruct. Researchers using this method must carefully assess corroboration and the potential for motivated distortion.

Section 8: Evaluating Research You Encounter

Claims about propaganda and disinformation circulate widely — in news articles, policy briefs, think-tank reports, advocacy materials, and social media. Being a careful consumer of this research is as important as being able to conduct it. When you encounter a research claim, apply the following questions.

Key Evaluation Questions

Sample and generalizability: - How large is the sample, and how was it recruited? A study of 120 university students in a single country cannot support claims about "how people" respond to propaganda. - Is the sample nationally representative, or is it a convenience sample? Online panels are common but introduce selection biases.

Methodology and causal claims: - What method was used, and is it appropriate for the claim being made? A correlation study cannot establish causation. A content analysis cannot tell you about effects. - Are controls adequate? Do the authors address alternative explanations for their findings?

Peer review status: - Has the research been peer-reviewed, or is it a preprint (posted to a repository before review)? Preprints are common in fast-moving fields and are not inherently unreliable — but they have not been subjected to independent expert scrutiny. - Preprint servers like SSRN, OSF, and arXiv are widely used in social sciences and computer science. Treat preprints as promising but preliminary.

Funding and conflict of interest: - Who funded the research? Industry-funded research on media effects — particularly research produced by or for platform companies — should be read critically. Disclosure of funding sources is standard in reputable journals; its absence is a red flag. - Does the research organization have an advocacy position that might shape its interpretation of findings?

Replication status: - Has the finding been replicated by independent researchers? Single studies, even well-designed ones, can produce false positives.

The Replication Crisis Context

Social psychology and communication research have been affected by a broader "replication crisis" — the finding that a substantial proportion of published results fail to reproduce when other researchers attempt them. A prominent example in propaganda-adjacent research is the Nyhan and Reifler backfire effect: their 2010 paper suggested that corrections can sometimes strengthen misbeliefs in people who hold them strongly. This finding was widely cited and shaped disinformation communication strategy. Subsequent replication attempts by multiple teams found little to no evidence of the effect under standard conditions. The lesson is not that research is worthless, but that individual findings — particularly surprising or counterintuitive ones — should be treated as hypotheses requiring further confirmation rather than established facts.

Red Flags in Research Reporting

Press release research: Studies publicized through press releases before peer review, particularly those reporting dramatic or counterintuitive findings
Industry-funded research without independent replication: Platform companies, think tanks funded by political interests, and advocacy organizations produce research, some of which is rigorous and some of which is not
Small-n studies generalized broadly: "A study of 50 participants shows that..." should prompt immediate questions about generalizability
Missing methodology: Credible research reports its methods in enough detail that another researcher could attempt replication. Methodology omitted or vaguely described is a serious concern

Section 9: Research Ethics in Propaganda Studies

IRB Requirements and the Limits of Institutional Review

Institutional Review Boards (IRBs) exist to protect human research participants from harm. For propaganda and disinformation research, IRB review raises specific questions:

Does exposing participants to propaganda or disinformation content in an experiment constitute harm, particularly if it might shift their beliefs?
Are participants fully informed about the purpose of the research, or does deception (common in psychology experiments) compromise meaningful consent?
Are the benefits of the research proportionate to the risks to participants?

The Facebook emotional contagion experiment (2014), discussed in Chapter 34, is the canonical cautionary case: Facebook and Cornell researchers manipulated the news feeds of nearly 700,000 users to test whether emotional valence of content affected emotional expression — without meaningful consent and without IRB oversight appropriate to the scale and nature of the intervention. The research was conducted as a platform product test rather than regulated academic research, allowing it to circumvent the protections IRB review is designed to provide. The episode prompted substantial debate about research ethics in platform-mediated environments.

Studying Extremist Communities Without Amplifying Them

Researchers studying extremist propaganda, disinformation ecosystems, and conspiracy theory communities face a genuine tension: detailed description of these communities can publicize their existence, normalize their discourse, or provide a roadmap for recruitment. Responsible practice involves:

Considering whether the research benefits of detail are proportionate to potential amplification risks
Avoiding publication of specific URLs, usernames, or recruitment materials that could serve as pathways into extremist communities
Consulting with affected communities and platform trust-and-safety teams where appropriate

Privacy in Network Analysis

Network analysis typically uses data collected at scale from platforms, sometimes without individual users' awareness that their public posts are part of a research dataset. Even when data is technically public, users may not have anticipated that their posts would be aggregated and analyzed. Responsible network researchers anonymize individual accounts in publications, avoid publishing data that could be used to identify or target individuals, and are transparent about data collection methods.

Responsible Disclosure in Influence Operation Documentation

Researchers who document ongoing influence operations — identifying specific accounts, infrastructure, and tactics — must consider disclosure timing and process. Publishing findings while an operation is ongoing can cause the operators to shift tactics and erase evidence. Coordinating disclosure with platforms, journalism organizations, and (where appropriate) government agencies allows for maximal accountability while preserving evidentiary integrity. The Stanford Internet Observatory's disclosure practice, which typically involves coordinated release with platform action, has become a field standard.

A Note on Methodological Humility

Research in propaganda studies is accumulating rapidly — the field has grown enormously since 2016 — but it remains incomplete, contested, and uneven. Many of the most consequential questions remain open: How large are propaganda effects under real-world conditions? Do inoculation campaigns scale? What makes some communities resilient and others vulnerable? Confident claims in either direction should be treated with skepticism. The researchers most worth trusting are typically those most forthcoming about what their methods cannot show.

See Appendix B: Key Studies and Annotated Examples for specific research examples using each methodology discussed here. See Appendix D: Primary Sources Guide for access instructions for the archives referenced in Section 6.