Case Study 17.2: Facebook's Internal Research on Harmful Content Amplification

"We Know What We're Doing to People"

Overview

In October 2021, Frances Haugen — a former product manager at Facebook who had worked on its civic integrity team — disclosed tens of thousands of pages of internal Facebook documents to the Wall Street Journal, to Congress, and to a consortium of seventeen news organizations. The documents, which became known as the "Facebook Files" and subsequently the "Facebook Papers," constituted the largest and most significant corporate disclosure since the tobacco industry's internal documents were released through litigation in the 1990s.

The Haugen disclosures established, from Facebook's own internal research, that the company's executives had been repeatedly informed of significant harms generated by the platform's algorithmic design — harms to civic information quality, to electoral integrity, to adolescent mental health, and to the spread of health misinformation — and had repeatedly chosen not to make changes to the platform's core design because those changes would reduce engagement and therefore advertising revenue.

This case study examines what the documents showed, the pattern of knowledge and inaction they documented, the structural parallels to Big Tobacco's internal documents, and the legislative responses that followed.

Frances Haugen: Background and Motivation

Frances Haugen joined Facebook in 2019 as a product manager on the civic integrity team, a unit responsible for identifying and addressing ways in which Facebook's platform could threaten democratic processes. She had previously worked at Google and Pinterest, and had come to Facebook specifically because she wanted to work on election integrity — an area of obvious public importance given the 2016 election experience with Facebook-distributed disinformation.

Over her two years at Facebook, Haugen became increasingly alarmed by a pattern she observed: her team's research would identify a problem, propose a solution, and the solution would be rejected or deprioritized for business reasons. The civic integrity team was dissolved after the 2020 election, its functions distributed to other teams without dedicated resources or mandate.

Before leaving Facebook, Haugen systematically copied thousands of internal documents. She worked with a securities attorney to file complaints with the Securities and Exchange Commission, alleging that Facebook had made misleading public statements to investors about its handling of harmful content. She disclosed the documents to Wall Street Journal reporters and, subsequently, to a consortium of international news organizations.

On October 5, 2021, she testified before the Senate Commerce Subcommittee on Consumer Protection. The hearing was notable for the bipartisan intensity of the questioning — senators from both parties expressed alarm at what the documents showed — and for Haugen's systematic framing of Facebook's conduct as a choice rather than a failure.

"Facebook has demonstrated that it is incapable of holding itself accountable," Haugen testified. "Facebook needs to be changed. The only way this will happen is with regulation."

What the Documents Showed: Civic Integrity

The 2018 Algorithm Change and Its Aftermath

In 2018, Facebook made a major change to its News Feed algorithm. The stated rationale was to prioritize "meaningful social interactions" — content that generated active engagement (comments, shares, replies) over passive consumption (likes, reactions, scrolling past). The change was presented publicly as a move toward healthier, more socially valuable content.

Internal documents told a more complex story. Before the change was implemented, Facebook's own researchers had warned that the new algorithm would systematically reward "engagement bait" — content designed to generate arguments, outrage, and reactive sharing rather than genuine connection. The researchers were correct: post-change analysis found that the algorithm rewarded content that provoked intense emotional reactions, which correlated strongly with political outrage and misinformation.

Internal documents also showed that Facebook's growth team understood this dynamic. The company tracked a metric it called the "MSI ratio" (meaningful social interactions), and internal analysis showed that the content types with the highest MSI scores were also the content types most associated with divisive political content and health misinformation.

The "Angry" Reaction and Algorithmic Amplification

One of the most consequential disclosures in the Haugen documents concerned the weighting Facebook assigned to different reaction types. When Facebook introduced the expanded reaction buttons (Love, Haha, Wow, Sad, Angry) in 2016, it faced a question: how should each reaction type be weighted in its engagement optimization algorithm?

Internal documents revealed that Facebook's algorithm assigned the "Angry" reaction approximately five times the weight of the "Like" reaction in its News Feed ranking. The rationale was empirical: the Angry reaction was a stronger predictor of subsequent engagement (continued time on platform, additional interactions) than the Like reaction. Users who reacted with Angry were more likely to continue engaging; from the algorithm's perspective, Angry was therefore a stronger signal of content quality.

The practical consequence was that content provoking anger received dramatically more algorithmic distribution than content provoking approval or passive appreciation. Political content that outraged users — content using fear-threat framing, enemy construction, and catastrophizing — was structurally advantaged in Facebook's distribution system. This was not an unintended side effect that Facebook's engineers failed to anticipate; it was a deliberate design choice made on the basis of engagement data. Internal researchers had flagged it as a problem. It was not changed.

Integrity Team Findings and Business Overrides

The documents revealed a systematic pattern in Facebook's internal governance: integrity research teams would identify a harm generated by a platform design choice, propose a change, and be overruled by business teams who argued the change would reduce engagement.

In one documented instance, Facebook's data science team proposed changes to the "People You May Know" recommendation system after identifying that it was connecting users to groups promoting health misinformation and vaccine skepticism. The changes were not implemented because they were projected to reduce Facebook group memberships, which was a key growth metric.

In another instance, integrity researchers documented that the "Groups You May Like" recommendation feature was systematically recommending users to increasingly extremist groups — following a pathway similar to YouTube's content recommendation logic, but applied to communities rather than videos. Proposed changes to reduce this effect were deprioritized for business reasons.

The documents showed that Facebook's integrity researchers were often highly competent, rigorous, and genuinely alarmed by their findings. They were not failing at their jobs; they were producing accurate research. They were then being told, through the organizational process of proposal review and resource allocation, that the business implications of acting on their research were unacceptable.

What the Documents Showed: Instagram and Adolescent Mental Health

Separate from the civic integrity documents, Haugen disclosed internal Facebook research on Instagram — Facebook's photo-sharing platform primarily used by younger users — that had reached alarming conclusions about the platform's effects on adolescent mental health.

The Slide Deck That Wasn't Published

In 2019, Facebook's internal research team completed a study on Instagram's effects on teenage girls' body image and mental health. The research was methodologically rigorous by the standards of corporate internal research: it involved surveys, in-app behavioral data, and qualitative interviews. Its findings were presented internally in a slide deck that became one of the most widely cited documents in the Haugen disclosure.

The research found that Instagram caused significant harm to the mental health of a substantial percentage of teenage girls. Key findings included:

• 32% of teenage girls who reported feeling bad about their bodies said that Instagram made those feelings worse.
• Teenagers who reported experiencing suicidal thoughts attributed their experiences to Instagram at rates significantly higher than to other online platforms.
• Instagram's architecture — specifically its emphasis on comparison through images and its algorithmic surfacing of idealized body images — drove social comparison dynamics that had well-documented negative mental health effects.

The company knew this. The research was not preliminary or speculative. It was well-executed internal research reaching clear conclusions. Facebook's decision was not to change Instagram's core design.

The Public-Private Gap

The gap between Facebook's public statements and its internal knowledge was particularly stark in the case of Instagram and adolescent mental health. Facebook's executives, including CEO Mark Zuckerberg, testified before Congress and made public statements claiming that Instagram was neutral or beneficial for most users' mental health and that concerns about its effects on teenagers were not supported by research.

These statements were made after — and in some cases after the company had become aware of — the internal research described above. The gap between what Facebook's executives said publicly and what the company's internal research showed was the basis for Haugen's SEC complaint and for multiple senators' characterizations of the company's conduct as fraudulent.

The Big Tobacco Structural Parallel

The most analytically significant framing of the Haugen disclosures — both in her Senate testimony and in subsequent academic and journalistic analysis — was the comparison to the tobacco industry's internal documents.

The Tobacco Pattern

Beginning in the 1950s, the major American tobacco companies conducted or commissioned internal research that established, to their own scientists' satisfaction, that cigarettes caused cancer and were addictive. The internal documents — which were not revealed to the public until litigation in the 1990s forced their disclosure — showed that the companies knew the harm they were causing and made deliberate decisions to deny it publicly, to fund external research designed to manufacture scientific uncertainty, and to suppress or discard internal research that reached inconvenient conclusions.

The Brown & Williamson document, disclosed in 1994, contained the most famous line in the entire corpus: "Doubt is our product, since it is the best means of establishing a controversy in the public's mind." This was an explicit articulation of a deliberate strategy to use manufactured uncertainty to delay regulatory action.

The Facebook Parallel

The structural parallel to Facebook is analytically significant, though it requires careful delineation. The parallel is not that Facebook's harms were equivalent in scale or type to tobacco's harms. It is not that Facebook's executives were engaged in the same deliberate suppression of research. The parallel is structural:

1. Internal research identified significant harm from the product.
2. Executives were informed of the research.
3. Business decisions were made not to change the product in ways that would mitigate the harm.
4. Public representations about the product's safety did not reflect what the internal research showed.

This is the pattern that defines institutional denial of harm — across industries, across eras. The specific harm, the specific product, and the specific mechanism vary; the institutional pattern does not.

Ingrid Larsen, presenting this parallel to her seminar, was careful about the limits of the comparison: "Tobacco executives knew their product would kill the people who used it and actively concealed that knowledge. That's a stronger case than Facebook, where the harms are more diffuse and the internal documents show genuine uncertainty as well as deliberate evasion. But the structural pattern — knowing-and-not-acting — is the same. And the tobacco history shows that 'structural similarity' is legally and morally significant even when the specific harms differ."

The Legislative Response

The Haugen disclosures generated significant legislative activity on multiple continents, though the outcomes differed substantially across jurisdictions.

United States: Senate Hearings and Stalled Legislation

Following Haugen's Senate testimony, several legislative proposals advanced in Congress. The most prominent was the Kids Online Safety Act (KOSA), which proposed requiring platforms to take reasonable measures to mitigate harms to minors, including harm from features that encouraged excessive use or promoted content associated with negative mental health outcomes.

KOSA advanced through committee consideration but did not pass into law during the 117th Congress (2021–2022). Critics argued that its requirements were too vague and could be used to restrict legitimate speech; supporters argued that the vagueness was inherent in the difficulty of regulating product design. Subsequent Congress sessions saw continued legislative activity without resolution.

At the state level, several states enacted legislation addressing social media and minors, including California's Age-Appropriate Design Code (2022), which drew directly on both the Haugen disclosures and the UK's Children's Code.

The EU Digital Services Act

The most comprehensive legislative response to the structural harms documented by Haugen and by years of academic research was the European Union's Digital Services Act, which entered into force in November 2022 and became fully applicable to very large online platforms in February 2024.

The DSA's approach to algorithmic harms is worth examining in detail as a regulatory model:

Systemic Risk Assessment. Very large online platforms (those with more than 45 million monthly active users in the EU) are required to conduct annual assessments of systemic risks arising from their services. Systemic risks specifically include: (a) the dissemination of illegal content; (b) actual or foreseeable negative effects for the exercise of fundamental rights; (c) actual or foreseeable negative effects on civic discourse or electoral processes; and (d) actual or foreseeable negative effects relating to public security or public health.

Mitigation Measures. Platforms that identify systemic risks are required to implement "reasonable, proportionate mitigation measures." The DSA does not prescribe specific measures — it does not, for example, require platforms to use chronological feeds or to apply specific algorithmic weightings — but it requires that measures be documented, that they actually address the identified risks, and that the assessment and mitigation process be subject to independent audit.

Data Access for Research. The DSA requires very large platforms to provide vetted researchers with access to data necessary to conduct independent assessments of systemic risks. This addresses a fundamental limitation in the academic research discussed in this chapter: most researchers have been forced to conduct their analyses using public-facing data (scraped content, API data) rather than the internal data that would allow more rigorous analysis. Under the DSA, researchers can conduct the kind of analysis that Haugen's disclosures showed was being done internally at Facebook.

Algorithmic Transparency. Users of very large platforms must be offered a recommender system option that is not based on profiling — in practical terms, a chronological or content-based feed that does not use behavioral data to personalize content. This does not eliminate engagement-optimized recommendation; it provides users with a genuine alternative.

The Significance of the DSA Model

The DSA's importance as a regulatory model is that it addresses the structural problem identified by the Haugen disclosures: Facebook's internal research identified systemic risks but there was no external requirement to act on that research. The DSA creates exactly such a requirement.

It does so without prescribing specific algorithmic designs, without requiring government review of content decisions, and without creating a government role in determining what content is true or false. It requires a process — systematic risk assessment, documented mitigation, independent audit — rather than a specific outcome. Whether this process model will be effective depends significantly on the quality of implementation, particularly on whether the audit and enforcement mechanisms are robust enough to distinguish genuine risk mitigation from compliance theater.

Implications for Propaganda Analysis

The Haugen disclosures and the Facebook case more broadly have several specific implications for the analytical frameworks developed in this course:

The channel shapes the message. The chapter's recurrent theme — that the medium shapes the message, in McLuhan's terms — is directly instantiated by the algorithmic amplification of outrage-generating content. Facebook did not cause specific propaganda messages to be created. But its algorithmic architecture systematically amplified messages with specific characteristics — emotional intensity, outrage, us-versus-them framing, threat — because those characteristics generated engagement. The channel did not merely transmit propaganda; it selected for it.

Corporate knowledge creates moral responsibility. The distinction between an architecture that happens to amplify propaganda and one that continues to do so after internal research has documented the effect is morally significant. Before Haugen, one could argue that Facebook's amplification of divisive political content was an unintended consequence of engagement optimization. After Haugen, that argument cannot be sustained for the period following the internal research findings. The company knew, at least in a probabilistic sense, what its design choices were doing.

Institutional interests shape information environments. The Haugen disclosures are a case study in how institutional interests — specifically, the revenue imperative that makes engagement optimization non-negotiable for an advertising-supported platform — shape the information environment available to the public. Facebook's information environment was not designed as propaganda infrastructure. But when designing for engagement conflicted with designing for civic information health, engagement won consistently. This is not a conspiracy; it is the predictable behavior of an institution following its own structural interests.

Discussion Questions

1. The Haugen comparison to Big Tobacco has been both widely adopted and disputed. Identify the strongest case for the comparison and the strongest case against it. Does the comparison help or mislead analysis of the Facebook situation?

2. Facebook's integrity researchers produced accurate and alarming research that was repeatedly not acted upon. What does this suggest about the relationship between good internal research and institutional change? Under what conditions does internal research produce institutional change, and under what conditions does it not?

3. The DSA's systemic risk assessment model requires platforms to identify and mitigate risks to "civic discourse" and "electoral processes." How would you operationalize these requirements? What would count as evidence that a platform had successfully mitigated risks to civic discourse, and who would determine this?

4. The "dark ad" problem — that behavioral micro-targeting allows different political messages to be shown to different audiences with no external visibility — is not addressed by the DSA's recommendation transparency requirement (which allows users to see their own recommendations but does not reveal what other users are seeing). Design a regulatory approach that would address the dark ad problem. What tradeoffs would your approach involve?

5. The Big Tobacco resolution — involving massive litigation, major financial settlements, significant regulatory action, and ultimately a dramatic decline in smoking rates — took approximately forty years from the first documented internal research to meaningful resolution. Does the trajectory of technology regulation suggest a similar timeline, a shorter one, or a longer one for resolving the harms documented by Haugen? What factors make the cases similar or different in this respect?

Case Study 17.2 | Chapter 17: Algorithms, the Attention Economy, and Filter Bubbles