Chapter 38: Deepfakes, Computational Propaganda, and Influence Operations

Part Seven: Emerging Frontiers

"I knew it was wrong. But I still felt like I saw it happen."

— Sophia Marin, seminar discussion

Sophia Marin had done her reading before class. She had worked through the assigned material, thought carefully about what she wanted to say, and arrived at the seminar prepared. Then Professor Webb asked how everyone had been thinking about the material since last week, and Sophia said: "I want to tell you something that happened to me."

She had been scrolling through her social media feed three days earlier — late evening, she noted, which she later recognized as relevant — when a video appeared. It showed a political figure she respects, someone she follows and has written about in her coursework. In the video, the figure was saying something they would never say. Something that contradicted almost everything they had publicly stood for. The video was clear, well-lit, and the voice sounded right.

She watched it three times.

It was only when she scrolled to the comments that she saw multiple people pointing out it was a deepfake. A good one, several commenters noted. She went looking for confirmation, found it, and felt an odd combination of relief and discomfort that she struggled to articulate.

"The relief was obvious," she told the seminar. "But the discomfort is what I keep coming back to. I knew it was wrong before I watched it the first time. Something in me knew. But I still watched it twice more, and by the third time, I could feel the emotion in my chest that I would have felt if it had been real. And afterward, I kept thinking: if I'm sitting here, knowing this person well, knowing their record, knowing their values — and I still felt like I saw it happen — what is happening to people who don't know them, who don't have that context, who encounter the video first and the debunking never?"

Ingrid Larsen said quietly: "That is exactly the research question."

This chapter begins there — in that gap between knowledge and feeling, between what we know to be true and what we experience as witnessed. That gap is where deepfakes operate. Understanding why requires us to examine both the technology and the psychology; both the operational mechanics of state-sponsored influence campaigns and the centuries-old tradition of manipulating visual evidence that they extend.

The psychology is worth pausing on before the technology, because the technology is designed to exploit it. When we watch a video of a person speaking, the experience is immediate and whole: we see a face, hear a voice, observe the micro-expressions and postural cues that we have been evolutionarily prepared to read from infancy. The credibility assessment happens before the analytical one — before we ask "is this authentic?", we have already felt the experience of witnessing. Propaganda systems have always worked by getting content into that pre-analytical space; the specific innovation of deepfake technology is that it extends the reach of fabrication into the domain of visual witness, which prior generations of propagandists could manipulate only through cruder means — edited footage, composite imagery, staged performances.

The Nazi Propaganda Ministry's Leni Riefenstahl directed Triumph of the Will (1935) as a masterpiece of staging political spectacle for the camera — manufacturing the visual experience of power and popular support from meticulously organized theatrical choreography. The ministry's newsreel division edited footage of world events for international distribution, removing context, reordering sequences, and occasionally splicing in unrelated material to support preferred narratives. What they could not do was manufacture a credible video of an actual opponent making an actual statement the opponent never made. Deepfake technology closes that gap. The logical endpoint of the ministry's approach — if you could generate any visual evidence you needed — was technologically constrained throughout the analog era. It is constrained no longer.

Tariq Hassan, who has been doing additional research since Chapter 37, had a folder of material open on his laptop. "I want to add something to what Sophia said," he said. "Because this isn't hypothetical. There are active, documented networks doing this at scale right now."

Prof. Marcus Webb nodded. "Then let's make sure we understand both the threat and the evidence."

38.1 What Deepfakes Are: A Technical Foundation

The term "deepfake" entered public discourse in late 2017, when a Reddit user of that name began posting synthetic videos that used artificial intelligence to superimpose one person's face onto another person's body in video footage. The technology beneath those early videos — and the far more sophisticated tools that have followed — rests on a class of machine learning architecture called a Generative Adversarial Network, or GAN.

A GAN works through competition. Two neural networks are trained simultaneously: a generator that produces synthetic images or video, and a discriminator that tries to distinguish between real and synthetic output. The generator improves by learning to fool the discriminator; the discriminator improves by learning to identify the generator's artifacts. Over hundreds of thousands of training cycles, this adversarial process pushes both networks toward their theoretical limits: a generator that can produce output indistinguishable from reality, and a discriminator that has developed sophisticated criteria for detection. When researchers then deploy the generator alone, they have a system capable of creating highly convincing synthetic imagery.

By the early 2020s, diffusion models — a different generative architecture based on the mathematics of thermodynamic processes — had begun to supplement and in some applications surpass GANs. Where GANs generate images in a single forward pass, diffusion models work by starting with pure noise and iteratively refining it toward a target. This produces output with exceptional photorealism and fewer of the characteristic artifacts that early GAN-based deepfakes exhibited.

For video deepfakes specifically, the most relevant application is face-swapping: replacing the face of a person in existing video footage with the face of another person, or replacing the mouth and lip movements of a speaker so that their image appears to say something it never said. The latter technique, sometimes called a "face-reenactment" or "talking head" synthesis, has become particularly refined. A system trained on sufficient footage of a person can animate their likeness to produce virtually any statement, in their voice, with their characteristic facial expressions and speaking patterns.

Voice cloning has developed in parallel and is in some respects ahead of video synthesis in terms of consumer accessibility. Systems available in 2024 can produce a convincing vocal clone from as little as three seconds of sample audio. The resulting synthetic voice captures not only the pitch and timbre of the original but also the speaker's rhythmic patterns, pronunciation habits, and emotional inflection. Combined with video synthesis, voice cloning enables the creation of audiovisual deepfakes in which both the image and the voice are synthetic.

The Collapse of the Expertise Barrier

The 2017 deepfakes required significant computational resources, expertise in machine learning, and time. By 2019, open-source tools had reduced the technical barrier substantially. By 2022, consumer applications — some operating through simple mobile interfaces — could produce face-swap deepfakes with no technical knowledge required. By 2024, text-to-video systems from major AI companies could generate synthetic video from written descriptions, without requiring any existing footage of the target.

This trajectory matters enormously for understanding the propaganda threat. In the early years of deepfake technology, the concern was primarily about what well-resourced actors — states, organized criminal groups — could do. The operational constraint was expertise. As that constraint has collapsed, the population of actors capable of producing propaganda-grade deepfakes has expanded from dozens to millions.

What Current Tools Can Produce

By 2024–2025, consumer-accessible deepfake technology could reliably produce:

• Face-swap videos of individuals speaking statements they never made, convincing under casual viewing conditions and in compressed social media formats
• Voice clones capable of deceiving people familiar with the subject, particularly in audio-only contexts (phone calls, voice messages)
• Composite imagery placing individuals in contexts they never occupied
• Animated "headshots" producing video speech from still photographs
• Audio recordings of statements never made by a target, in their voice

What current technology cannot reliably produce:

• Long-form video that maintains coherence across many minutes without artifacts detectible by careful review
• Video that survives expert forensic analysis without detectable manipulation signatures
• Content that can defeat all available detection systems simultaneously
• Material that is fully resistant to authentication methods that verify source provenance

This qualification matters. The public discourse around deepfakes often overstates current capabilities, creating a perception that all video is now equally untrustworthy. That perception is itself a propaganda-relevant effect — an outcome we will examine when we discuss the "liar's dividend."

It is also worth noting what the trajectory of capability development means for the chapter's analysis. The cases discussed throughout — the Gabon President video, the Slovak election audio, the Indian electoral deepfakes — represent the state of deployed operational deepfakes at specific moments. The generation quality of deployed deepfakes typically lags behind the frontier of what is technically achievable, because operational deployment involves considerations (time to produce, available footage of the target, acceptable artifact levels for the intended distribution context) that research demonstrations do not. But the gap between research frontier and operational deployment has been narrowing. What is technically challenging today will be operationally routine tomorrow; analysis calibrated only to current capability will be retrospectively inadequate within a short horizon.

38.2 The Taxonomy of Synthetic Media Threats

Deepfakes are not a single threat category but a family of related threats with different mechanisms, different target populations, and different relationships to propaganda. Careful analysis requires distinguishing between them.

(a) Identity Theft Deepfakes

The most directly propaganda-relevant category: using a person's likeness without consent to produce false statements attributed to them. A politician appears to announce a policy reversal. A CEO appears to confirm a financial fraud. A journalist appears to recant a story. A military official appears to order a ceasefire or, more dangerously, an escalation. The mechanism is simple: the deepfake exploits the credibility and authority of the target to lend that credibility and authority to a false claim.

(b) Non-Consensual Intimate Imagery (NCII)

By volume, the most prevalent use of deepfake technology is the creation of sexual imagery depicting real individuals without their consent. Studies published between 2019 and 2023 consistently found that the large majority of deepfake videos in circulation were NCII targeting women, predominantly celebrities. This is not the primary propaganda focus of this chapter, but it is directly relevant to the information environment in two ways: first, NCII deepfakes of political figures have been used as a weapon of political intimidation, particularly targeting women in public life; second, the normalization of non-consensual deepfake creation in one domain shapes the cultural acceptance of deepfakes in others.

(c) Political Manipulation Deepfakes

The category of greatest interest to this textbook: deepfakes that specifically target political figures, institutions, or processes to influence political opinion, discredit opponents, create false documentation, or undermine trust in democratic processes. These can range from the relatively crude (a politician appearing to make racist statements) to the operationally sophisticated (synthetic media designed to be "discovered" as evidence in an unfolding political crisis, calibrated to specific target audiences in specific political moments).

(d) Fraud Deepfakes

Voice cloning and video deepfakes are increasingly used for financial fraud: impersonating corporate executives to authorize fraudulent transfers (CEO fraud), impersonating family members to request emergency money, and creating false endorsements for financial products. In early 2024, a finance employee at a Hong Kong company transferred $25 million USD to fraudsters who conducted an entirely deepfake video conference call, with synthesized images of the company's CFO and other senior executives. This category is not primarily propaganda but shares the infrastructure and capability base with political manipulation deepfakes, and the economic incentives driving its development accelerate the capability improvements that make political deepfakes more effective.

The Propaganda Threat Hierarchy

Across these categories, the greatest propaganda threat is not the production of convincing individual deepfakes but rather the use of deepfake capabilities in combination with computational amplification, targeting, and the exploitation of what researchers call the "liar's dividend." A single deepfake, however sophisticated, can be debunked. A deepfake integrated into a coordinated influence operation, distributed across multiple platforms through networks of amplifying accounts, timed to a politically sensitive moment, and later used to seed doubt about authentic evidence — that combination is qualitatively different, and it is what the documented state-actor operations described in this chapter have attempted to construct.

38.3 The "Liar's Dividend" in Detail

In a foundational 2019 paper, law scholars Bobby Chesney and Danielle Citron introduced the concept of the liar's dividend: the strategic benefit that flows to bad actors not from producing deepfakes themselves, but from the existence of deepfake technology and the public awareness of that existence.

The argument is elegant and disturbing. If audiences know that convincing deepfakes exist, they become potentially susceptible to a new category of denial: any authentic video documentation of a politician, official, or public figure doing or saying something incriminating can be dismissed as a deepfake. "That video was fabricated." "That recording is AI-generated." "Deep state disinformation." The truthfulness of the dismissal is irrelevant; what matters is its plausibility in an environment where such fabrications are known to be possible and where most audiences lack the forensic tools to distinguish authentic from synthetic.

The Mechanism in Practice

The liar's dividend operates through a fundamental asymmetry. Creating doubt is cognitively and argumentatively cheaper than establishing truth. If authentic video footage of a political figure's misconduct exists, proving it authentic requires audiences to understand and trust a chain of verification — provenance, metadata, forensic analysis, multiple independent witnesses — that is complex, technical, and easily attacked by sophisticated adversaries. Claiming it might be a deepfake requires nothing except the observation that deepfakes exist.

This asymmetry is exploitable by any actor who benefits from the non-establishment of truth. Authoritarian states facing authentic documentation of human rights abuses are obvious beneficiaries. In 2023, footage of Russian forces appeared online depicting what credible journalists and human rights organizations identified as evidence of war crimes. Russian state media and affiliated accounts immediately described the footage as fabricated — "Western deepfakes," AI-generated propaganda. The claim did not need to persuade a majority to be effective; it needed only to create sufficient doubt in target audiences that the footage could not perform its evidentiary function.

The Malaysian Politician Case

One of the earliest and most clearly documented examples of the liar's dividend in operation predates the modern deepfake era. In 2019, a sex tape emerged appearing to show a Malaysian politician. The politician denied its authenticity, claiming the footage had been fabricated to damage his reputation. Independent forensic analysts reached conflicting conclusions. The case became a political crisis not primarily because audiences believed the footage was authentic or fabricated, but because the existence of the controversy itself — the impossibility of certain determination — served the political interests of those who wanted to damage the politician's credibility while also allowing his allies to claim victimization.

The deepfake era accelerates this dynamic. What once required a genuine disputed claim now requires only the invocation of a technology whose existence is publicly known.

Democratic Accountability Implications

Prof. Webb identifies the liar's dividend as potentially one of the most significant long-term threats to democratic accountability. Democratic systems depend on the possibility of documentation: the ability to produce evidence of misconduct that audiences can accept as credible. Elections can be influenced by video of candidates. Criminal prosecutions of powerful figures require evidentiary documentation. Journalistic accountability journalism depends on the existence of records that cannot be plausibly denied.

"Consider what accountability journalism requires," Webb said in seminar. "It requires that when you show someone doing something, the audience can believe they're seeing something that happened. If that condition is compromised, accountability journalism is compromised. Not because deepfakes will fool journalists — journalists have access to authentication tools. But because the public watching the same journalism can now be told, by any motivated party, that the evidence itself cannot be trusted."

This is not merely hypothetical. It is the operational doctrine of the most sophisticated information operations documented in recent years.

38.4 State Actor Influence Operations: The Operational Architecture

Individual deepfakes or disinformation posts are not, in themselves, influence operations. The documented cases of effective state-sponsored information operations share a common operational architecture that distinguishes them from individual bad actors or opportunistic disinformation.

Objectives Beyond Persuasion

Classical propaganda theory, as this textbook has examined through its historical cases, primarily aims at persuasion: changing beliefs, motivating action, building loyalty. The documented objectives of state-sponsored influence operations are often different, and sometimes explicitly not persuasive. The goals more commonly described in the academic literature on Russian and Chinese operations are:

Confusion — producing multiple contradictory narratives about an event so that no single account can achieve dominant credibility. A military incident produces four different explanations simultaneously; audiences unable to sort through them default to uncertainty.

Polarization — not persuading audiences to a particular view but deepening existing divisions, amplifying fringe positions to make them appear more mainstream, and increasing the emotional temperature of existing conflicts. The documented Russian Internet Research Agency operation in 2016 was not primarily targeted at persuading centrist voters to support Trump; it was targeted at maximizing cultural and political division across the existing American political spectrum.

Erosion of trust — in media, in institutions, in the possibility of shared knowledge. The goal is not a particular belief but the meta-belief that reliable belief is impossible.

Information environment preparation — creating the conditions in which future disinformation will be more readily accepted, by seeding doubt about existing credible sources and establishing alternative credible sources that can later be used.

These objectives represent a more sophisticated approach to information operations than simple persuasion, and they are more difficult to counter because they do not depend on any particular claim being believed. The operation succeeds when audiences stop trying to determine what is true.

Organizational Structure

State-sponsored influence operations typically involve multiple layers:

At the center are professional content producers and strategic planners, often connected to state intelligence or military structures. Their role is target identification, narrative design, and content production. These are not trolls operating on personal initiative but employees with operational objectives, targets, and performance metrics.

The amplification layer consists of networks of inauthentic accounts — fake personas, hijacked accounts, coordinated real-person accounts — whose role is to distribute content, manufacture apparent consensus, and boost organic content that serves the operation's objectives.

A peripheral layer engages unwitting participants: real people who share content from the operation's network without knowing its origin, because the content has been designed to appeal to genuinely held beliefs and frustrations.

The Propaganda Ministry's International Operations as Precedent

This architecture has historical precedent. Joseph Goebbels' Propaganda Ministry in Nazi Germany was not solely focused on the domestic German audience. It maintained active international operations: funding foreign-language publications, cultivating relationships with sympathetic foreign journalists, producing content that appeared to be the organic output of foreign political movements but was in fact designed in Berlin.

The propaganda ministry's film division engaged in systematic manipulation of newsreel footage for international distribution — editing authentic footage to remove context, splicing composite images into documentary records, and producing fake "documentary" evidence of purported enemy atrocities or German military achievements. These techniques were understood as distinct from the more obviously theatrical internal propaganda, precisely because they were designed for audiences who needed to believe they were seeing unmanipulated documentary evidence. The goal was to produce the impression of authentic documentation in foreign audiences who did not already believe the message.

The ministry also maintained what was known as the Auslandsorganisation — the Foreign Organization of the Nazi Party — which cultivated networks of sympathetic individuals in foreign countries who would distribute German propaganda while appearing to be organic participants in their national political life. The parallel with the IRA's network of fake American personas and Spamouflage's amplification of authentic diaspora voices is not accidental. The strategic logic — using apparent insiders to deliver messages that would be dismissed if attributed to their true source — is identical. What has changed is the scale at which it can be executed, the speed at which content can be produced and distributed, and the ability to target content to specific population segments with precision the analog era could not approach.

State-sponsored digital influence operations are the contemporary version of this structure — with the key difference that the distribution infrastructure is global, instantaneous, and algorithmically amplified.

Prof. Webb made the historical connection explicit in seminar: "The reason we study the Nazi Propaganda Ministry is not because it was uniquely evil in its methods — it wasn't; many states conducted similar operations. It's because it's the best-documented case of a fully professionalized, systematic state information operation, operating both domestically and internationally, across multiple channels, with clear organizational structure and measurable objectives. The IRA and Spamouflage are that — built on modern infrastructure, at global scale, but recognizably the same kind of thing."

38.5 Spamouflage and Chinese Influence Operations

The term Spamouflage was coined by researchers at Graphika and later used jointly with the Stanford Internet Observatory to describe a specific network of coordinated inauthentic behavior linked to the Chinese state. The name combines "spam" (high-volume, automated, low-quality content distribution) with "camouflage" (the attempt to make artificial content appear organic).

Operational Discovery

The Spamouflage network was first documented publicly in 2019, when Graphika published a report identifying a coordinated network of Facebook, Instagram, and YouTube accounts producing content favorable to the Chinese Communist Party and critical of pro-democracy figures in Hong Kong. The accounts were characterized by several operational signatures: newly created or recently repurposed personas with sparse posting histories, high posting volumes, cross-platform coordination using identical or near-identical content, and posting patterns inconsistent with human behavior (activity continuing through overnight hours without breaks, mechanically uniform posting intervals).

Subsequent reports — notably the Stanford Internet Observatory's 2022 Spamouflage analysis — documented the network's expansion and evolution. By 2022, the operation had grown to encompass thousands of accounts across Facebook, Twitter, YouTube, TikTok, Reddit, Pinterest, Tumblr, and a number of smaller platforms. Meta's transparency reporting began specifically identifying Spamouflage as a recurring removal target, with quarterly takedowns removing hundreds to thousands of accounts per reporting period.

Content and Targeting

The Spamouflage operation's content strategy has evolved but maintains consistent thematic priorities:

Promoting the Chinese state's narrative on sensitive issues: Xinjiang (characterizing accounts of detention and abuse as fabrications, circulating Chinese government talking points), Hong Kong (delegitimizing pro-democracy protesters, describing the movement as Western-funded instigation), Taiwan (promoting reunification narratives, delegitimizing the Taiwanese government), and the COVID-19 pandemic origins (circulating Chinese government-aligned narratives about Western responsibility).

Discrediting democratic institutions and processes: Content targeting Western electoral systems, security forces, and governance institutions with the goal of reducing trust and amplifying existing grievances.

Targeting Chinese diaspora communities: Perhaps the most operationally sophisticated targeting involves Chinese-speaking communities outside China — in Australia, Canada, the United States, Europe — who exist in an information space where Chinese-language content is available but content moderation in Chinese languages has historically lagged behind English.

Amplifying genuine voices: The most sophisticated evolution of the operation involves identifying genuine Western voices who express views aligned with Chinese state objectives and amplifying their content through the Spamouflage network. This is operationally more valuable than producing synthetic content because it uses authentic first-person credibility while providing the same amplification function.

The Computational Dimension

What distinguishes Spamouflage from earlier Chinese influence operations — and from the Russian IRA model — is its explicitly computational character. The IRA built networks of fake personas with elaborate backstories, posting histories, and social relationships; its approach was intensive in human labor. Spamouflage prioritizes volume through automation: large numbers of accounts, algorithmic content variation to evade detection, rapid account replenishment as takedowns remove existing accounts.

This computational approach reflects a different strategic calculation. The IRA model aimed for quality: creating fake American personas so convincing that they built genuine audiences and relationships. The Spamouflage model, at least in its operations targeting Western audiences, appears to prioritize reach and confusion over persuasion: flooding the information environment with pro-China content, normalizing those narratives through sheer volume, and making it difficult for audiences to assess what is grassroots opinion and what is artificial amplification.

Tariq Hassan, presenting his research to the seminar, put it directly: "The Stanford Internet Observatory data shows quarterly takedowns running into the hundreds of thousands of accounts. Each takedown is a snapshot — by definition what's been caught. The question isn't whether this is happening. The question is what it's designed to accomplish and whether it's working."

38.6 Russian Operations Post-2016

The Russian Internet Research Agency's 2016 operation has been extensively documented — most authoritatively in the Senate Intelligence Committee report and the indictment of IRA-linked individuals by the Mueller investigation. What is less publicly discussed is the degree to which Russian influence operations adapted and evolved after 2016, in ways that made them in some respects harder to detect.

The IRA's 2016 Approach and Its Limits

The IRA's 2016 operation was built around elaborate fake American personas: activists, community organizers, political commentators who had built genuine followings over months and years of authentic-seeming engagement. The operation's strength was the credibility of its fake personas; its weakness was the investment required to build those personas and the investigative surface those personas created when exposed.

After 2016, as platform detection methods improved and the IRA's methods became publicly documented, Russian operations shifted emphasis.

Post-2016 Evolutions

Amplification of authentic fringe voices: Rather than creating fake personas, Russian-linked operations increasingly identified genuine American commentators, activists, and media outlets whose existing views aligned with Russian objectives and amplified their content through coordinated networks. These voices could not be "taken down" as inauthentic because they were authentic; the manipulation was in their amplification.

The Secondary Infektion operation: Documented by the EU DisinfoLab and subsequently confirmed by the Stanford Internet Observatory, Secondary Infektion was a Russian influence operation that operated from at least 2014 through 2020 across more than 300 platforms. Unlike the IRA, which focused on social media, Secondary Infektion specialized in placing content in smaller forums, comment sections, and websites where it would appear to be organic discussion, then circulating screenshots of that "discussion" as evidence of broader opinion. The operation's distinctive signature was the use of authentic leaked documents — occasionally mixed with fabricated documents designed to appear alongside authentic ones — to create maximum confusion about what was real.

Integration with RT and Sputnik: Russian state media channels, particularly the international-facing RT (formerly Russia Today) and Sputnik networks, serve a dual function: producing content for direct consumption by international audiences and serving as laundering infrastructure for influence operation narratives. A narrative seeded in social media by fake accounts can be amplified by RT, giving it a "journalistic" source that can then be cited by other outlets.

The Ukraine Information War: Russia's February 2022 invasion of Ukraine produced the most intensive documented information operations of the modern era, with multiple overlapping objectives: justifying the invasion domestically (denazification, NATO expansion, protecting Russian speakers), deflecting international documentation of atrocities, and managing the domestic information environment in countries Russia was trying to pressure into reduced support for Ukraine. This operation incorporated deepfake elements, including a documented deepfake of Ukrainian President Zelensky appearing to call on Ukrainian soldiers to surrender, which was rapidly identified and debunked but widely circulated before correction.

The Ukraine case is particularly instructive because it represents the first large-scale test of whether a targeted democracy could fight a simultaneous kinetic war and information war with significant success. Ukraine's response to the Zelensky deepfake was rapid: within hours, Zelensky appeared in an authentic video filmed from what was clearly identifiable as downtown Kyiv, directly refuting the fake. This prebunking-adjacent response — getting authentic counter-content into the same channels before the deepfake could consolidate its effect — represents one of the more successful rapid responses to a political deepfake on record. The success was enabled by the specific circumstances: a known target, immediate motivation to respond, and an audience already primed (by the context of active invasion) to scrutinize the claim's plausibility.

Ingrid Larsen placed the Ukraine information war in the Nordic context: "Finnish and Swedish researchers have been studying Russian information operations since at least 2007 — the Estonian cyberattacks were the first major case. What they've documented is a consistent pattern of escalation: each operation is more sophisticated than the last, each one tests different vulnerabilities, and the institutional knowledge accumulates. The Ukraine operation was the most sophisticated to date. The next one will be more sophisticated than that."

38.7 Coordinated Inauthentic Behavior: The Platform Concept

Meta's policy team developed the concept of Coordinated Inauthentic Behavior (CIB) to provide a workable framework for what their trust and safety operations could detect and act on, distinct from content-based moderation.

The CIB Definition

CIB is defined as the use of fake or misleading account information to coordinate multiple accounts toward a common inauthentic goal. The key elements are: coordination (multiple accounts acting together), inauthenticity (the accounts or their operators misrepresent themselves), and behavior (the focus is on the pattern of action, not the content). This framing deliberately separates platform action from content judgments: CIB enforcement does not require Meta to determine whether content is true or false, only whether the accounts distributing it are behaving authentically.

The distinction has significant practical and principled advantages. It allows platform enforcement action that is agnostic about political content — a coordinated fake network promoting views from any part of the political spectrum can be removed without the platform making editorial judgments. It also focuses enforcement on the manipulation of the information environment rather than on specific claims, which aligns with the operational reality that sophisticated influence operations change their content frequently while maintaining consistent behavioral signatures.

Platform Transparency Reports

Meta, Twitter (pre-Musk acquisition), and YouTube have all published CIB transparency reports documenting specific takedowns, their geographic origins, and their operational characteristics. These reports, together with the independent analyses published by Stanford Internet Observatory and Graphika, constitute the primary empirical record of documented influence operations.

These reports have important limitations. They document what platforms found and removed, which is a subset — of uncertain size — of what was operating. Detection methods are never fully disclosed, for the reasonable reason that disclosure would enable adversary evasion. Report frequency and comprehensiveness have varied considerably across platforms and time.

Ingrid Larsen observed a pattern relevant to this analysis: "The Nordic countries have invested significantly in studying these operations — there are research centers in Stockholm, Copenhagen, and Helsinki that work specifically on Russian influence operations in Northern Europe. And one thing they consistently find is that the takedown reports undercount the actual activity. The operational intent isn't to escape detection. It's to operate at a scale where what gets caught doesn't significantly reduce the overall effect."

What Detection Reveals and Conceals

CIB detection operates primarily through behavioral signals: unusual account activity patterns, shared infrastructure (IP addresses, device fingerprints), coordinated posting timing, template-based content variation, network structure analysis. These signals are most reliable for catching unsophisticated operations and operations that prioritize volume over detectability.

The most sophisticated operations are specifically designed to minimize these signals: using legitimate devices and connections, spacing activity to appear human, maintaining genuinely distinct content across accounts, building genuine (if shallow) engagement histories. The detection ecosystem is therefore best understood as a filter that catches a proportion of influence operations — likely catching more of the less sophisticated — while allowing sophisticated operations longer operational lives.

38.8 Political Deepfakes in Practice

The Gabon President Video (2019)

In January 2019, President Ali Bongo of Gabon released a video address to the nation after an extended public absence due to health concerns. His appearance and manner in the video prompted immediate speculation that it might be a deepfake. Experts who analyzed the footage were divided; some identified visual artifacts consistent with deepfake manipulation, others concluded it was authentic footage of a man recovering from a stroke.

What is significant for propaganda analysis is not the resolution of the authenticity question — which remains genuinely contested — but the way the deepfake possibility was weaponized in the subsequent political crisis. Members of the military cited the possibility that the video was fabricated as partial justification for a coup attempt the following week. The liar's dividend operated in real time: the existence of deepfake technology was invoked to delegitimize a real or genuine communication and to provide narrative cover for an unconstitutional seizure of power.

Slovak Election Deepfakes (2023)

In September 2023, two days before Slovak parliamentary elections, audio recordings circulated on social media that appeared to show Michal Simecka, leader of the progressive party PS, discussing how to buy votes from the Roma community. The recordings spread rapidly. Slovak fact-checkers and the PS campaign identified them as deepfakes within hours, but Slovak electoral law prohibits campaigning in the 48 hours before an election, which meant that formal responses and advertising refuting the claims could not be distributed through normal channels during the period of maximum exposure.

Simecka's party lost the election. Subsequent analysis concluded the timing had been operationally deliberate: releasing a deepfake in the blackout period maximized exposure time while minimizing the target's ability to respond. This represents a specific operational innovation — the use of electoral blackout periods as a deployment window for disinformation.

Indian Election Deepfakes (2024)

The 2024 Indian general election produced the largest documented scale of electoral deepfake deployment. Multiple parties deployed deepfake videos of their own leaders — creating synthetic speeches in regional languages to reach voters in areas where the leader had not campaigned. Deepfakes of deceased leaders were produced endorsing current candidates. Opposition deepfakes attributed statements to governing party leaders designed to damage their credibility.

The scale and multi-directional nature of the Indian 2024 deepfake deployment represents a qualitative shift from isolated operations to pervasive synthetic media in electoral contexts. It suggests a possible future trajectory in which deepfakes are normalized as electoral tools used by all political actors, making the standard assumption of video authenticity untenable in any electoral information environment.

The Indian case also raises a specific complication for the regulatory debate. When deepfakes are used by multiple competing parties — including some with arguably legitimate electoral purposes like regional-language voter outreach — the case for prohibition becomes more complex. If deepfake video of a leader speaking a language they do not actually speak is used to reach voters in a linguistically diverse democracy, the same technology enables both fraud and genuine political communication. This is not a hypothetical edge case: the Indian electoral commission received complaints about both offensive deepfakes (attributing false statements to opponents) and more benign ones (allowing national leaders to speak to local communities in their own languages) in the same electoral cycle. Regulatory frameworks that sweep both into a prohibition must grapple with this ambiguity, and the difficulty of that grappling helps explain why comprehensive electoral deepfake law has been slow to emerge even in democracies motivated to address the problem.

The Jordan Peterson Case

The specifics of the Jordan Peterson deepfake case — involving a synthetic video of the psychologist and political commentator appearing to make statements contrary to his documented views — are analyzed in detail in Case Study 01. For the purposes of the chapter discussion: the case illustrates the specific propaganda mechanism of identity theft deepfakes used against politically controversial figures. Peterson's existing public profile created both the target (his credibility and authority with specific audiences could be weaponized against those audiences) and the complication (his audiences' familiarity with his actual views made the deepfake more likely to be recognized). The case thus illustrates both the power and the limitation of deepfakes targeting high-profile, well-documented figures.

38.9 Detection and Authentication: The Current State

Forensic Detection Methods

Deepfake detection research has produced a growing toolkit of forensic methods for identifying synthetic video:

Visual artifact analysis: Early deepfakes exhibited characteristic artifacts — inconsistent facial boundaries, unnatural eye blinking patterns (early models trained on photographs tended not to include blinking), background distortion near the face boundary, inconsistent skin texture at the margins of the synthesized area. As generation models have improved, these artifacts have become subtler and less reliable as detection signals.

Frequency domain analysis: Synthetic images often produce distinctive patterns in frequency-domain representations that differ from authentic photographs and video. These patterns are invisible in normal viewing but detectable through Fourier transform analysis. As detectors have published research on these patterns, generation models have been retrained to minimize them — a characteristic cat-and-mouse dynamic.

Physiological signals: Authentic video of humans contains subtle physiological signals — the slight variations in skin color associated with blood flow (remote photoplethysmography, or rPPG), the micro-movements of breathing, the involuntary pupil responses to lighting changes. Deepfake video currently fails to replicate these signals accurately, making them potentially useful forensic indicators. Researchers caution that as generation models are trained on this research, these signals will become targets for synthesis.

Metadata and provenance analysis: Authentic recordings contain metadata that records when, where, and how they were captured. This metadata can be forged, but forensic examination of metadata chains, combined with analysis of compression artifacts and encoding characteristics, can identify video that has been processed or altered in ways inconsistent with its claimed history.

Why Detection is a Losing Race

The fundamental structural problem with deepfake detection is that detector research is public and generative model training is private (or at least faster-adapting). When researchers publish a new detection method, that publication describes, in precise terms, a weakness in current generation models. Model developers — including those building tools for influence operations — can use that published research to identify and eliminate the weakness in next-generation models. Detection is therefore always calibrated against yesterday's generation capability.

This dynamic has been consistently observed in the deepfake detection literature. Detection models trained on deepfakes from 2019 perform poorly on 2022-vintage deepfakes. Detection models trained on 2022-vintage deepfakes struggle with 2024-vintage deepfakes. There is no theoretical reason to expect this dynamic to reverse.

The Authentication Alternative

Given the structural limitations of detection, researchers including Hany Farid (one of the leading deepfake forensics researchers) have argued for reorienting the response around authentication rather than detection: rather than trying to prove that a video is fake, develop systems that allow authentic videos to prove their authenticity.

The Coalition for Content Provenance and Authenticity (C2PA) has developed a technical standard for this approach. C2PA-compatible cameras embed a cryptographic signature in media at the moment of capture, creating a verifiable record of when, where, and how the content was created. If this signature is present and unbroken, the content can be verified as authentic; if it is absent, the content is unverified — not necessarily fake, but without provenance guarantees.

The limitation is adoption: C2PA is only useful if cameras, platforms, and audiences participate in the system. In 2025, the standard is implemented in a small number of professional cameras and is beginning to appear in smartphone photography. Its coverage of the global content ecosystem remains limited.

Digital watermarking — the embedding of invisible signals in AI-generated content to identify its synthetic origin — has been mandated in some jurisdictions and adopted voluntarily by several major AI companies. The technical limitations of watermarking are significant: current watermarks can be removed or degraded through simple image processing, and there is no reliable watermarking approach that survives determined adversarial removal.

38.10 Regulation of Deepfakes

The United States

The United States has no comprehensive federal deepfake legislation as of 2025. Congressional proposals have repeatedly stalled, partly due to First Amendment concerns and partly due to the definitional challenges described below. Several states have enacted deepfake-specific laws:

California: AB 602 (2019) permits individuals depicted in deepfake pornography to sue creators. AB 730 (2019) criminalizes the distribution of deepfakes of political candidates within 60 days of an election. Additional legislation in 2023 extended the political deepfake provisions and addressed synthetic media in advertising.

Texas: HB 4337 (2023) prohibits deepfake sexual images and deepfakes intended to deceive voters about a candidate, with criminal penalties.

Virginia: Extended existing revenge porn statutes to cover synthetic imagery in 2019.

This state-level patchwork creates inconsistent protection and enforcement that is particularly weak against foreign actors.

The European Union AI Act

The EU AI Act (2024) requires that AI-generated and AI-manipulated content be clearly labeled, with specific provisions for synthetic media depicting individuals. The Act's approach focuses on transparency requirements — disclosure of synthetic origin — rather than prohibition, reflecting both the definitional challenges of prohibition and the political economy of regulating technology in a trade-focused regulatory environment. The Act does impose stricter requirements on high-risk AI applications, which include some election-adjacent uses of generative AI.

China's Deepfake Regulation

China enacted the "Provisions on Deep Synthesis Internet Information Services" in 2023 — one of the more comprehensive national deepfake regulatory frameworks in existence. The regulations require labeling of AI-generated content, restrict the generation of "fake news" using deepfake technology, and establish that service providers are responsible for ensuring their platforms are not used for deepfake disinformation.

The existence of this regulation creates a studied analytical irony that Tariq Hassan highlighted in seminar: "China regulates deepfakes domestically — among the tightest regulations on synthetic media of any country — while the Spamouflage operation runs coordinated inauthentic behavior in foreign information environments. The regulation and the operation serve the same interest: the CCP controlling the information environment, domestically through prohibition, internationally through exploitation."

This pattern — regulatory control over domestic information plus active offensive use of information operations abroad — is consistent across the documented record of Chinese state information policy and represents a specific form of information sovereignty doctrine.

The Definition Problem

One consistent barrier to effective deepfake legislation is the difficulty of defining what a "deepfake" is in legally actionable terms. Definitions that are too narrow (requiring machine learning synthesis) exclude non-AI manipulation techniques that produce similar effects. Definitions that are too broad risk capturing satirical and artistic uses of digital manipulation that have a long legitimate history in journalism and political commentary.

The deepfake of former President Trump placed in various scenarios by political opponents; the satire deepfakes used by late-night television programs; the documentary reconstructions that use actor performances to represent historical figures — all of these involve synthetic media and digital face manipulation. A prohibition broad enough to capture political manipulation deepfakes is likely broad enough to capture some of these legitimate uses as well.

38.11 Research Breakdown

Goldstein and Grossman (2023): "How Disinformation Evolved"

Joshua Goldstein and Renée DiResta's 2023 analysis (with Shelby Grossman) in Foreign Affairs provides one of the more rigorous frameworks for understanding how AI capabilities intersect with influence operations. The analysis distinguishes between three types of AI-enabled operations:

Quantity amplification: Using AI to produce large volumes of content for existing influence operation infrastructure — content farms producing at higher volumes, astroturfing networks with more diverse content.

Quality enhancement: Using AI to improve the persuasiveness, targeting precision, and cultural calibration of influence operation content.

Novel capability: Operations that could not be conducted without AI — the creation of convincing synthetic video of real individuals being the clearest example.

The paper's key finding is that at the time of writing, most documented AI-enabled influence operations were primarily using AI for quantity amplification, not for the more sophisticated quality enhancement or novel capability applications. The bottleneck was not technology but operational sophistication: generating large volumes of convincing synthetic content is still not the hardest part of an influence operation. Understanding the target information environment, selecting the right narratives, and distributing content effectively within that environment remain the limiting factors.

This finding has important implications for how resources are allocated in the response to AI-enabled influence operations. If the current threat is primarily quantitative rather than qualitatively novel, the most important interventions may be those that address content volume rather than deepfake sophistication specifically.

Graphika and Stanford Internet Observatory Joint Reports

The collaborative research program between Graphika and the Stanford Internet Observatory has produced a series of joint reports on specific influence operations. Their methodological contribution is the integration of network analysis (mapping relationships between accounts, identifying coordination signatures) with content analysis (understanding narrative objectives and targeting strategies) and provenance analysis (tracing content back to origins).

A consistent methodological finding across their reports: the most detectable influence operations are not the most dangerous ones. The Spamouflage network, despite its scale and longevity, shows limited evidence of successfully shifting public opinion in its target communities. Its continued operation may reflect a different success criterion: not persuasion but normalization, saturation, and the maintenance of an operational infrastructure that can be redirected as strategic needs change.

38.12 Primary Source Analysis: The Template Evidence

Among the most analytically revealing artifacts from documented influence operations are the content templates and coordination guides that have been recovered through platform investigations, legal proceedings, and investigative journalism. These materials allow analysis not just of what operations produced but how they were organized and what their operators understood their objectives to be.

The IRA Operational Templates

The Mueller investigation's document production included recovered IRA operational documents that specified, in considerable detail, the operational parameters for the U.S. targeting operation. These documents described target audiences using American political categories (conservatives, African Americans, LGBT communities), specified content themes and emotional registers for each target audience, established posting schedules and platform-specific strategies, and set performance metrics (engagement rates, follower counts, cross-posting to other accounts in the network).

The documents reveal an operation with a sophisticated understanding of American political culture and social media dynamics. The content guidance for targeting African American communities, for example, specifically addressed voter demobilization — not persuading Black Americans to vote Republican, but discouraging participation through content designed to deepen cynicism about the electoral system. The content guidance for conservative targeting addressed cultural grievances and law enforcement themes.

Applying the propaganda framework from earlier chapters: these operational documents represent a sophisticated integration of target audience analysis, emotional appeals, and channel selection. What is distinctive is the explicit demobilization objective — the IRA understood that reducing participation by some populations was as strategically valuable as increasing enthusiasm among others. This is an application of information operations theory derived from Cold War psychological operations doctrine, updated for social media platforms.

The demobilization targeting is worth dwelling on because it illustrates a strategic logic that standard propaganda analysis often misses. Most propaganda theory focuses on conversion: moving audiences from one belief to another, or from neutrality to conviction. Demobilization targeting operates through a different mechanism: it works by intensifying existing cynicism, deepening disengagement, and making the argument (through emotional rather than logical means) that political participation is futile. The Big Tobacco parallel is relevant here too: the tobacco industry's manufactured scientific controversy did not primarily need to persuade anyone that smoking was safe; it needed only to prevent the consensus finding from achieving actionable public conviction. The IRA's voter demobilization content did not need to persuade anyone that democracy was illegitimate; it needed only to raise the emotional cost of engagement enough to suppress turnout in specific populations.

Sophia connected this to her opening-chapter concern: "At the beginning of this course, I said I was worried about what was happening in my family's living room — my parents watching things I couldn't verify, forming opinions I didn't recognize. I've been thinking about this all semester. And what I realize now is that the most dangerous thing isn't a specific lie. It's the feeling that nothing can be trusted. Because if nothing can be trusted, you stop trying to figure out what's true. And if you stop trying, you stop participating. And if you stop participating, whoever is manufacturing the feeling of untrustworthiness wins without having to convince you of anything."

Prof. Webb said: "That is, with considerable precision, the firehose of falsehood doctrine. We'll take it up directly in Chapter 39."

Big Tobacco Parallel: Manufacturing Consensus

The Big Tobacco industry's decades-long effort to create the impression of scientific uncertainty about smoking's health effects offers an analytical parallel to computational consensus manufacturing. Internal documents recovered in litigation (and now publicly available through the University of California San Francisco's Industry Documents Library) show a deliberate strategy of identifying sympathetic researchers, funding studies designed to produce ambiguous results, amplifying marginal dissenting voices, and creating front organizations that appeared to represent grassroots scientific concern.

The structural parallel to contemporary influence operations is precise: in both cases, the objective is not to persuade audiences of a positive claim but to prevent them from accepting the consensus claim. The mechanism is not fabricating evidence but flooding the information environment with contradictory signals until confident knowledge formation becomes difficult. The digital influence operations documented in this chapter can deploy this strategy at scales and speeds that the tobacco industry's research-funding approach could not approach.

38.13 Debate Framework: Should Deepfakes of Public Figures Be Prohibited?

Position A: Targeted Prohibition is Necessary and Constitutional

The core argument for targeted prohibition of political deepfakes holds that certain synthetic media — specifically, realistic video or audio falsely depicting a public figure making a verifiable statement — causes direct, demonstrable harm to democratic processes that cannot be adequately addressed through existing legal frameworks or non-legal means.

The harm is demonstrable: the Slovak election deepfake, the Indian election deepfake campaigns, the Zelensky surrender deepfake, and the dozens of other documented cases establish that political deepfakes have been deployed in real electoral and conflict contexts with real political effects. The inability to detect them reliably means existing defamation remedies are inadequate: by the time a deepfake is proven false and remedies pursued, the electoral moment has passed.

A narrowly targeted prohibition — focused on realistic synthetic media depicting verifiable statements by public figures in a political context, with explicit carve-outs for satire, comedy, and clearly labeled artistic works — is achievable and has precedent. The distinction between realistic simulation and obvious parody has been navigated in existing law. The standard of "a reasonable viewer would believe this depicts an actual event" is workable.

The First Amendment concern is real but overstated. The Supreme Court has consistently held that false statements of fact receive limited First Amendment protection, and realistic synthetic media attributing specific false statements to real people sits squarely in this category. The incidental burden on protected satire is manageable through careful drafting.

The argument that prohibition will not work against state actors is true but irrelevant to the domestic case. Domestic bad actors — political campaigns, partisan operators — can be deterred and punished. The existence of state actor violations does not remove the value of prohibiting private actor violations.

Position B: Non-Legal Responses are Necessary and Prohibition Will Fail

The core argument against prohibition holds that legal prohibition of deepfakes of public figures cannot be drafted narrowly enough to avoid chilling legitimate expression, will face sustained constitutional challenge, will not be enforceable against the primary threat (state actors), and diverts resources from responses that might actually work.

The definitional problem is fatal in practice. The line between "realistic simulation" and "obvious parody" is not objectively determinable; it depends on audience, context, and distribution environment. A deepfake that is obviously satirical to one audience is consumed as authentic by another. Legal frameworks cannot impose a single audience on content that circulates across populations with different political literatures, media sophistication, and exposure to the original.

The constitutional barrier is more serious than the pro-prohibition argument acknowledges. Courts have not consistently extended reduced First Amendment protection to all false statements of fact; the jurisprudence is complex and contested. And a law specifically targeting content involving public figures' likenesses in political contexts sits at the intersection of multiple highly protected categories.

The enforcement gap is dispositive for the primary threat. State-sponsored influence operations — the Spamouflage networks, the Russian operations — are entirely indifferent to U.S. domestic law. A prohibition regime in which the primary beneficiaries of deepfakes are legally unreachable and the primary costs fall on legitimate domestic expression has inverted its priorities.

The responses that might work are technical (C2PA authentication infrastructure, which can verify authentic content without prohibiting anything) and educational (prebunking/inoculation campaigns that build audience resilience to deepfake manipulation). These approaches do not require defining deepfakes legally, do not face constitutional barriers, and can be effective against all actors rather than only domestic private ones.

38.14 Action Checklist: Evaluating Suspicious Video and Audio

Sophia's experience — watching a deepfake three times while knowing something was wrong — illustrates that intuition alone is not a reliable detection method. But neither is technical sophistication a realistic demand to make on general audiences. The following checklist offers realistic steps that ordinary media consumers can take.

Step 1: Pause before sharing. The operational design of influence operations exploits the shareability of emotionally resonant content. The first response to any video that generates strong emotion — particularly outrage or anxiety about a trusted political figure — is to pause before sharing it.

Step 2: Check the source. Where did this video come from? What is the original platform account that posted it? Is that account established, with a posting history consistent with its stated identity? New accounts, newly repurposed accounts, or accounts with unusual activity patterns are indicators worth noting.

Step 3: Reverse video search. Google's reverse image search and tools like InVID/WeVerify allow frame-by-frame reverse searching of video content. This can identify whether the video has appeared before in a different context, whether identical footage is being used with a new narrative, or whether the same faces appear in other videos with different apparent identities.

Step 4: Lateral reading. Do not try to evaluate the claim in the video from within the video. Search for coverage of the same claim in established news sources. If a major political figure has made a major statement, it will be covered by multiple independent sources; if only one source has the video and no established outlets have picked it up, that is a significant indicator.

Step 5: Check the claim against what you know. Does this statement align with the speaker's documented record, positions, and rhetorical style? Significant divergence from well-documented patterns is a flag. (This is the step Sophia's knowledge allowed her to perform intuitively.)

Step 6: Look for visual artifacts — with calibrated expectations. Under careful review, current deepfakes often exhibit: unusual blurring or pixelation at the boundary of the face; inconsistent lighting between face and background; unnatural smoothness of skin texture; eye movements that are too regular or too irregular; lip synchronization that is slightly off in timing or in the visible mouth articulation. But: sophisticated deepfakes may not exhibit obvious artifacts. The absence of visible artifacts is not proof of authenticity.

Step 7: Recognize the limits of your detection ability. Individual audiences cannot reliably detect high-quality deepfakes under normal viewing conditions. The goal of personal verification practice is not to become a forensic analyst but to slow down the sharing of potentially manipulated content and to route uncertain cases to professional fact-checkers and authentication tools before they propagate through your network.

Step 8: Consider the timing. Influence operations are designed to deploy in politically sensitive moments — close to elections, during ongoing crises, in periods of heightened anxiety. Content that arrives with unusual urgency in a politically charged moment deserves additional scrutiny.

38.15 Inoculation Campaign: Deepfake Threat Assessment

Progressive Project — Future-Proofing Component

Chapter 37 asked you to assess the specific threat AI-generated text poses to your target community's information environment. This component extends that analysis to visual and computational threats.

The Assessment Framework

Part 1: Current Threat Inventory

What synthetic media threats are currently documented as active in information environments relevant to your community? Use platform transparency reports, Stanford Internet Observatory reports, and Graphika takedown analyses to identify:

• Are there documented influence operations targeting the political or cultural issues your community cares about?
• Are there documented operations targeting communities demographically or geographically similar to your target community?
• Have any deepfake incidents occurred in your target community's electoral or political context in the past three years?

Part 2: Awareness Assessment

What does your target community currently understand about deepfake technology and computational influence operations?

• Does your community have baseline awareness that deepfakes exist and that political deepfakes have been used in real elections?
• Does your community understand the liar's dividend — that authentic video can be falsely dismissed as deepfake?
• What is the community's likely response to a well-crafted political deepfake? (Consider the emotional resonance of the specific political figures who might be targets.)

Part 3: Inoculation Message Design

Building on your prebunking framework from earlier chapters, design an inoculation message specifically addressing deepfake threats.

The message should: - Preemptively expose the manipulation technique (face synthesis, voice cloning) in accessible terms - Provide the emotional inoculation — acknowledging that even knowing a deepfake exists, viewers often experience it as felt reality — so that the experience is anticipated rather than disorienting - Give the community specific, actionable steps from the verification checklist - Address the liar's dividend explicitly: authentic video can be falsely called a deepfake, so absence of authentication does not mean fabrication

Part 4: Computational Threat Integration

If Spamouflage-style computational amplification is relevant to your community's information environment, address the coordinated amplification threat separately from the deepfake threat. These require different inoculation messages: the deepfake message addresses the authenticity of specific content; the CIB message addresses the manufactured appearance of consensus.

Deliverable: A 600–800 word brief analyzing your target community's deepfake and computational influence operation threat landscape and the inoculation messages you would deploy. This brief feeds directly into your Capstone Inoculation Campaign document.

38.16 The Authentication Alternative: Provenance as Defense

The dominant response to the deepfake problem has been detection — developing tools and training audiences to identify fabricated content. But detection-first strategies carry a structural weakness that grows more severe as generation technology improves: every detection method is a race against a moving target, and the generator is always one step ahead of the verifier. This asymmetry has led researchers, technologists, and policymakers toward a fundamentally different question. Rather than asking how do we prove this is fake, they ask how do we prove this is real?

This reframing — from detection to authentication — defines the provenance approach to information integrity. Instead of analyzing content after the fact for signs of manipulation, provenance systems embed verifiable records of origin and custody into media files at the moment of creation. The logic is architecturally inverted: rather than hunting for anomalies that signal falsity, audiences and platforms can check for the presence of a verified chain linking content to a known source.

The C2PA Standard

The most developed implementation of this approach is the Coalition for Content Provenance and Authenticity (C2PA), an open technical standard developed by a consortium of technology and media companies — including Adobe, Microsoft, Sony, and Nikon — and released publicly in 2022. The C2PA standard specifies how cryptographic metadata can be embedded directly into media files: photographs, videos, and audio recordings. This metadata records the device that captured the original file, the software used in any subsequent editing, any modifications made during post-production, and the identity (if disclosed) of the creator or publisher.

The metadata is cryptographically signed. This means that any alteration of the content — whether cutting frames, adjusting audio, or wholesale replacing faces — breaks the signature, producing what the standard's architects call a "tamper-evident seal." The file itself becomes a carrier of its own provenance record, a ledger of custody that travels with the content wherever it is shared or republished.

Adobe has implemented C2PA under the commercial brand "Content Credentials," visible as an icon — sometimes called the "cr" badge — that users can click to inspect a piece of media's provenance record. Camera manufacturers including Nikon have begun shipping hardware that signs images at the moment of capture, before any editing has occurred, which provides the strongest possible chain of custody: the record begins at the sensor.

Content Credentials in Practice

The content credentials concept is most powerful when understood as a chain of custody rather than a simple origin stamp. Each step in a media file's lifecycle — from capture to edit to publication — can be logged in the provenance record. A photograph taken by a news agency can carry credentials showing it was captured on a specific verified device, edited in an approved workflow, and published by a credentialed newsroom. That record does not prevent manipulation of the underlying pixels, but it does mean that any manipulation breaks a verifiable seal, transforming an ambiguous authenticity question into a traceable audit trail.

For audiences, this represents a meaningful shift. Instead of attempting their own content analysis — scrutinizing lip synchronization, checking for digital artifacts, consulting reverse-image search tools — users could in principle rely on a credential indicator analogous to a website's HTTPS lock: not proof of truth, but proof of traceable origin.

The Limits of Provenance

The authentication approach, however, carries its own structural vulnerabilities that honest analysis requires acknowledging. The most fundamental is the adoption problem: provenance infrastructure is only useful if it is universal. A credentialing system that covers thirty percent of media production does not tell audiences that uncredentialed content is fake — it only tells them that uncredentialed content lacks a credential. In an environment where most content is uncredentialed, the absence of a credential carries no diagnostic value.

Metadata can also be stripped. A credentialed video downloaded from a reputable source and re-uploaded without its embedded metadata loses its provenance record entirely, producing a clean file that carries no indication it was ever authenticated. Social media platforms and messaging applications have historically stripped or compressed metadata during upload processing, which can inadvertently destroy provenance records even without malicious intent. Existing platform architecture would require deliberate redesign to preserve rather than discard these signals.

Provenance infrastructure also provides no help with historical footage. The vast archive of video and audio content already in circulation — news broadcasts, speeches, interviews, documentary footage — was captured before credential-embedding hardware existed. Deepfakes that appropriate historical imagery cannot be countered with a technology that only authenticates new content.

The Ecosystem Requirement

The provenance approach, if it is to operate at scale, requires what its proponents call a content authenticity ecosystem: camera manufacturers embedding signing hardware at the point of capture; editing software preserving and extending provenance records through post-production workflows; social media platforms passing rather than stripping metadata during upload and distribution; newsrooms and publishers implementing credentialing as a standard workflow step; and browsers or viewing applications surfacing credentials to end users in interpretable form. Each of these requirements involves organizational incentives, competitive dynamics, technical integration costs, and political considerations that do not resolve automatically because a standard exists.

Near the end of a seminar session on technical countermeasures, Sophia raised the question she said she kept coming back to: her grandmother watches a lot of video content online, shares things she finds compelling, and has neither the digital fluency to use forensic detection tools nor the inclination to seek them out. Would any of this actually help her?

Professor Webb paused before answering. "Honestly? Provenance technology could help your grandmother — but only if every platform she uses adopts it, only if it's surfaced in a way she can see and understand, and only if it's adopted universally enough that the absence of credentials is meaningfully suspicious rather than just normal. Right now, none of those conditions exist. The technology is real. The ecosystem isn't."

An Institutional Problem Wearing a Technical Mask

The provenance approach illuminates something important about the deepfake problem that purely technical framings tend to obscure. The challenge is not, at its root, a problem of detection capability or even authentication architecture. It is a problem of coordinated institutional adoption. The technology to embed and verify provenance exists. What does not exist is the regulatory framework that would create adoption incentives, the platform-level commitment to preserve rather than discard metadata, the camera manufacturer consensus to prioritize signing hardware, or the public-facing design work that would make credentials legible to ordinary users rather than specialists.

This is a pattern that recurs throughout the history of information integrity efforts: the technical solution is developed before the social infrastructure required to deploy it at scale. Detection tools, prebunking curricula, provenance standards, and platform labeling policies are all real contributions. But each operates at the level of the individual intervention while the problem operates at the level of an ecosystem. The deepfake threat, like the computational propaganda threat that accompanies it, is ultimately a stress test not of technical ingenuity but of institutional capacity — the capacity of distributed actors with competing interests to coordinate around shared norms before the information environment deteriorates past a threshold of recovery.

Chapter Summary

This chapter has examined the visual and computational dimensions of the emerging propaganda landscape — the dimensions that extend and amplify what Chapter 37 identified in AI-generated text.

Deepfakes are not the central threat in the current influence operation landscape; as the research breakdown noted, most documented AI-enabled operations are using AI for volume rather than for sophisticated deepfake production. But deepfakes are central to the logic of the emerging information environment in two ways that go beyond their direct use as propaganda.

First, through the liar's dividend: the existence of deepfake technology — even in the absence of specific deepfake deployment — creates a denial capability for anyone who benefits from rejecting authentic documentation. State actors committing human rights abuses, politicians caught in damaging footage, officials whose misconduct is recorded — all gain a new tool for undermining the evidentiary basis of accountability journalism, not by producing deepfakes but by invoking the category's existence.

Second, through the computational amplification infrastructure: the Spamouflage model, the evolved IRA model, the Secondary Infektion operation — these are not primarily deepfake operations. They are coordinated inauthentic behavior operations that can integrate deepfake content as one tool among many. The infrastructure exists independently of any specific technique and can incorporate new capabilities as they become available.

Sophia's experience — watching a video three times while knowing it was wrong, and still feeling it — is not evidence of personal credulity or insufficient media literacy. It is evidence of how human perceptual systems work: we are not built to evaluate video as documentary evidence with probabilistic skepticism. Our experience of watching a face say something is immediate and pre-analytical in ways that resist conscious correction. The propaganda systems being built today are explicitly calibrated to exploit that gap.

Prof. Webb, closing the seminar discussion, offered the frame that the chapter returns to: "The research on prebunking suggests that knowing about the technique in advance is protective. Not perfectly protective — there is still the visceral experience Sophia described. But it reduces the after-effect: the lingering sense that you saw something happen. If we can build environments where people encounter the concept of a deepfake before they encounter a deepfake, the technique loses some of its power."

"How much of its power?" Sophia asked.

"Enough to matter," Webb said. "Not all of it. But enough."

Key Terms

Deepfake — Synthetic media produced using machine learning (typically GANs or diffusion models) that depicts a real individual saying or doing something they never said or did.

Generative Adversarial Network (GAN) — A machine learning architecture in which two neural networks — a generator and a discriminator — compete, producing increasingly realistic synthetic output.

Voice cloning — The use of AI to produce a synthetic reproduction of an individual's voice from a sample of authentic recordings.

Liar's dividend — The strategic benefit that flows to bad actors from the existence of deepfake technology: the ability to dismiss authentic documentary evidence as potentially fabricated.

Coordinated Inauthentic Behavior (CIB) — Meta's framework for identifying and removing networks of accounts that coordinate using fake or misleading identities to achieve inauthentic objectives.

Spamouflage — A Chinese state-linked influence operation network characterized by high-volume, cross-platform, computationally automated content distribution.

C2PA (Coalition for Content Provenance and Authenticity) — A technical standard for embedding cryptographic authentication signatures in media at the point of capture, enabling verification of authentic content provenance.

Content farm — An operation (human or AI-augmented) that produces large volumes of content optimized for platform distribution, typically designed to generate engagement rather than inform.

Secondary Infektion — A long-running Russian influence operation documented operating from 2014–2020, characterized by placement of forged and authentic documents across multiple platforms and forums.

Face-reenactment synthesis — A deepfake technique in which the mouth and lip movements of a video subject are replaced with synthesized articulation corresponding to statements they never made.

Chapter 38 of Propaganda, Power, and Persuasion: A Critical Study of Influence, Disinformation, and Resistance

Previous: Chapter 37 — AI-Generated Content and the New Disinformation Landscape Next: Chapter 39 — Information Warfare and the Future of Truth