Chapter 19 Exercises: Fact-Checking Methods, Organizations, and Limitations

These exercises develop practical skills in fact-checking methodology, critical analysis of verification systems, and hands-on verification of sample claims. Complete each exercise as directed; some require original research, some require analysis, and several walk you through the full fact-checking process.

Part A: Concept Application Exercises

Exercise 19.1 — Fact-Checking vs. Opinion Checking

For each of the following statements, determine whether it is (a) a check-worthy factual claim, (b) a purely evaluative statement not amenable to fact-checking, or (c) a mixed statement containing both factual and evaluative components. For category (c), identify which portion is factual and check-worthy.

1. "The unemployment rate fell to 3.8 percent last month."
2. "The current administration has been the most corrupt in the history of this country."
3. "This policy will destroy the economy."
4. "Under the previous administration, the national debt increased by $7 trillion."
5. "Immigration from Central America increased 40 percent over the past two years, which proves that border policies have failed."
6. "Studies show that this vaccine is 95 percent effective at preventing severe illness."
7. "Our city's schools are the best in the state."
8. "The crime rate in this neighborhood increased significantly after the shelter was opened."

For each statement in category (c), write a revised version that isolates only the check-worthy factual claim.

Exercise 19.2 — Claim Selection Priority Ranking

You are the editor of a small fact-checking organization with two staff members and a budget for verifying approximately five claims per week. The following ten claims have been submitted by readers for fact-checking. Rank them by priority (1 = highest priority) and explain your reasoning for each ranking decision. Consider significance, reach, verifiability, and newsworthiness.

1. A U.S. Senator's claim that a proposed infrastructure bill will create 2 million jobs.
2. A local city council member's claim that a proposed zoning change will increase traffic by 15 percent.
3. A viral social media post claiming that a common over-the-counter pain reliever causes liver failure in 1 in 10,000 users.
4. A presidential candidate's claim that their opponent voted against a veterans' benefits bill "17 times."
5. A celebrity's tweet claiming that a specific food is linked to reduced cancer risk.
6. A claim, spreading via WhatsApp, that a particular neighborhood is being targeted for surveillance by unmarked police vehicles.
7. An op-ed claim that the United States spends more per capita on healthcare than any other developed country.
8. A talking head on cable news claiming that crime in the nation's cities is at an all-time high.
9. A nonprofit organization's fundraising email claiming that a species is on the verge of extinction.
10. A school board member's claim that test scores in their district improved 22 percent under their tenure.

Write a brief (2-3 sentence) justification for each ranking decision.

Exercise 19.3 — Rating Scale Design

Design your own fact-checking rating scale for a hypothetical organization. Your scale should:

1. Include between 4 and 7 distinct categories.
2. Have clear, operationally distinct criteria for each category.
3. Handle the special case of claims that are technically true but misleading in context.
4. Include an explanation of what distinguishes adjacent categories.
5. Address how your scale handles claims that are unverifiable given current evidence (vs. false).

After designing your scale, apply it to the following three claims (assume you have already conducted the verification research):

• A politician claimed that the GDP grew by 4.2 percent in the prior year. Official government data confirms the GDP grew by 4.2 percent in the calendar year cited, but the politician omitted that the figure was heavily influenced by a one-time statistical adjustment and that trend growth was 2.1 percent.
• A claim that "scientists have confirmed that drinking coffee reduces the risk of Alzheimer's disease by 30 percent." Several studies have found associations between coffee consumption and reduced Alzheimer's risk, but the scientific consensus characterizes this as preliminary evidence with significant methodological limitations; no study establishes causation.
• A politician's claim that they "never voted to cut Social Security." The politician has a 100 percent voting record against Social Security benefit cuts, but they did vote in favor of a budget resolution that included a proposal that would have required Social Security negotiations — a vote that opponents characterized as supporting cuts but which does not directly constitute a cut vote.

Exercise 19.4 — Analyzing the Truth-O-Meter Distribution

The table below shows the hypothetical distribution of PolitiFact ratings for two politicians over a given period:

1. Calculate the percentage of "mostly accurate or better" ratings (True + Mostly True) for each politician.
2. Calculate the percentage of "mostly inaccurate or worse" ratings (False + Pants on Fire) for each politician.
3. What would need to be true about the claim selection process to conclude that Politician B simply makes more false statements? What alternative explanations for the distribution difference should be considered?
4. A critic argues that PolitiFact is biased against Politician B because it selected more of their claims to check. A defender argues that PolitiFact is doing its job by checking more claims from the politician who makes more check-worthy statements. How would you evaluate these competing claims? What data would help resolve the dispute?

Exercise 19.5 — Analyzing a Published Fact-Check

Select a published fact-check from PolitiFact, FactCheck.org, or the Washington Post Fact Checker that was published within the past 6 months. After reading the fact-check in full, answer the following questions:

1. What specific claim is being checked? Write the claim in your own words.
2. What was the final rating, and which organization produced the fact-check?
3. What primary sources did the fact-checkers consult? List at least five sources referenced.
4. Did the fact-checkers consult expert sources? Who were they and what were their relevant credentials?
5. Is there any context that you believe the fact-checkers should have included that they did not?
6. Do you agree with the rating? If not, what rating would you have assigned and why?
7. Was the fact-check transparent about the limitations of its own analysis?

Part B: Hands-On Fact-Checking Exercises

Exercise 19.6 — Guided Fact-Check: Health Claim

Conduct a full fact-check of the following claim, following professional methodology:

Claim: "The United States has the highest rate of maternal mortality of any wealthy developed country."

Steps to follow: 1. Identify what data sources would contain authoritative information on maternal mortality rates across countries. 2. Locate at least two independent data sources that address this claim. 3. Note any methodological complexities in measuring and comparing maternal mortality across countries (how is "maternal mortality" defined? How are wealthy developed countries classified? What time period is most relevant?). 4. Write a 400-500 word fact-check in the style of a professional fact-checking organization, including a rating using the Truth-O-Meter or Pinocchio scale of your choice. 5. Note any limitations of your verification research.

Exercise 19.7 — Guided Fact-Check: Historical Claim

Conduct a full fact-check of the following claim:

Claim: "The New Deal's Works Progress Administration employed more than 8 million Americans at its peak."

Steps: 1. Locate the most authoritative historical sources on WPA employment data. 2. Determine the peak period of WPA employment and the figures associated with it. 3. Note any definitional or methodological complexity (what counts as "employed by the WPA"?). 4. Write a 300-400 word fact-check with a rating.

Exercise 19.8 — Guided Fact-Check: Statistical Claim

Conduct a full fact-check of the following claim:

Claim: "The richest 1 percent of Americans own more wealth than the bottom 90 percent combined."

Steps: 1. Identify the most authoritative data sources on wealth distribution in the United States (Federal Reserve, Census Bureau, academic economists' datasets). 2. Check the claim against available data, noting the most recent figures available. 3. Assess whether the claim's framing (1% vs. 90%) is accurate or could be more accurately stated. 4. Identify the time period to which the claim applies and whether it has been consistently true across time. 5. Write a 400-500 word fact-check with a rating.

Exercise 19.9 — Reverse Engineering a Fact-Check

Below is a set of evidence points that would be relevant to a fact-check. Without looking up the original claim, construct the fact-check that this evidence supports. Then assign a rating.

Evidence set: - Official FBI crime statistics show violent crime rates declined from 2015 to 2022 in the United States overall. - Several major cities (including New York City, Los Angeles, and Chicago) saw violent crime rate increases in 2020-2021, followed by partial declines in 2022. - The FBI changed its crime data reporting methodology in 2021, creating a gap in year-over-year comparisons. - Public polling data consistently shows that large majorities of Americans believe crime has increased, regardless of what official statistics show. - Perceptions of local crime are more accurate than national perceptions in most studies.

Write: 1. The most likely claim that prompted this fact-check. 2. Your fact-check narrative (300 words minimum). 3. Your rating and justification. 4. A brief note on what you would have checked if you had more time and resources.

Exercise 19.10 — The "Missing Context" Problem

The following claims are technically accurate but potentially misleading due to missing context. For each: (a) Explain what context is missing and why it matters. (b) Write a corrected version that includes the necessary context. (c) Assign a PolitiFact-style rating and explain whether "Mostly True" or "Half True" is more appropriate.

1. "Our state's high school graduation rate has reached a record 91 percent." (Context not provided: the graduation rate was calculated using a new, less stringent method introduced two years ago; under the previous method, the rate would be 84 percent.)
2. "Crime fell 15 percent in our city last year." (Context not provided: the city changed its crime classification system, reclassifying many incidents that previously counted as crimes; analysts estimate true crime fell 3-5 percent.)
3. "This country sends $X billion in foreign aid every year." (Context not provided: the stated figure is total foreign assistance budget; actual cash transfers are a fraction of this, with the majority being military equipment and technical assistance.)

Part C: Critical Analysis Exercises

Exercise 19.11 — The Partisan Perception Problem

Study design analysis: Researchers want to measure the extent to which partisans perceive fact-checkers as biased against their side. Design a study that would: 1. Measure the baseline partisan lean of participants. 2. Expose participants to fact-checks that are (a) favorable to their party, (b) unfavorable to their party, and (c) neutral in partisan valence. 3. Measure perceived bias in the fact-checker for each condition. 4. Include appropriate control conditions. 5. Address the ethical considerations of exposing participants to political content.

After designing the study, answer: What results would you expect based on the hostile media effect literature? What results would be surprising?

Exercise 19.12 — Evaluating the Scalability Problem

The scalability gap between misinformation volume and fact-checking capacity is a fundamental challenge. For this exercise:

1. Research the approximate volume of new content posted daily on Twitter/X, Facebook, and YouTube.
2. Research the approximate number of fact-checks published per week by the five largest professional fact-checking organizations combined.
3. Calculate the ratio of content volume to professional fact-check production.
4. Evaluate three potential approaches to closing this gap — expanding professional staffing, investing in AI automation, and expanding crowdsourced verification — assessing each against the dimensions of feasibility, effectiveness, accuracy, and cost.
5. Write a 500-word recommendation memo as if you were advising a large foundation deciding how to allocate a $50 million grant to improve the quality of public information.

Exercise 19.13 — Automated vs. Human Fact-Checking Comparison

Compare the capabilities of automated fact-checking systems (like ClaimBuster) against human professional fact-checkers across the following dimensions. For each dimension, explain which approach performs better and why:

1. Speed of processing
2. Scale of coverage
3. Contextual reasoning
4. Expert consultation
5. Handling of novel claims with no precedent in existing databases
6. Adversarial robustness
7. Transparency and explainability of decisions
8. Cost per fact-check

Based on your analysis, identify what division of labor between automated systems and human fact-checkers would be most effective.

Exercise 19.14 — Community Notes Design Critique

Community Notes uses a bridging-based ranking algorithm that requires cross-partisan agreement for notes to display. Analyze this design choice:

1. What problem does the cross-partisan requirement solve? Be specific.
2. What problems does it fail to solve or potentially create?
3. Design an alternative mechanism that achieves the same anti-partisan-bias goal through a different means.
4. What data would you want to collect to evaluate whether Community Notes is achieving its goals?
5. Research any published academic evaluations of Community Notes and summarize their findings (cite at least one peer-reviewed source).

Exercise 19.15 — Global Fact-Checking Challenges

You have been asked to advise a new fact-checking organization launching in a country of your choice (choose a country in Sub-Saharan Africa, South Asia, or Southeast Asia). Research the information environment in that country and write a 600-word advisory memo addressing:

1. The primary platforms through which news and information spread in that country.
2. The primary sources of misinformation in that country's context.
3. The availability of authoritative primary sources for fact-checking.
4. The legal and political environment for press freedom.
5. At least three methodological adaptations you would recommend relative to standard Western fact-checking practice.

Part D: Research and Writing Exercises

Exercise 19.16 — Historical Analysis of a High-Profile Fact-Check

Identify a major fact-check published between 2008 and 2020 that received significant media attention (examples: PolitiFact's 2013 "Lie of the Year" — "If you like your health plan you can keep it"; fact-checks of specific 2016 debate claims; various COVID-19 fact-checks). Write a 700-word retrospective analysis that addresses:

1. What claim was checked and what was the verdict?
2. Was the verdict accurate? Has subsequent evidence supported or challenged the original rating?
3. How was the fact-check received by partisan audiences? Can you find evidence of how people responded to it?
4. What was the fact-check's impact on public belief, political behavior, or the broader information environment?
5. What, if anything, would you do differently if you were conducting this fact-check today?

Exercise 19.17 — Comparing Organizational Methodologies

Select the same claim — choose a specific, verifiable factual assertion — and check how different fact-checking organizations handled it. You should find the claim checked by at least two different organizations (examples: climate statistics, economic figures, and historical claims are often checked by multiple organizations).

Analyze: 1. Did the organizations reach the same conclusion? 2. What sources did each organization consult? 3. Did they frame the claim similarly, or did framing choices differ? 4. If they reached different ratings, what accounts for the divergence? 5. What does this comparison suggest about the objectivity of fact-checking?

Exercise 19.18 — The Correction Decay Problem

The effectiveness of fact-checks may diminish over time as memories of the correction fade while the original false claim may be re-encountered repeatedly. This is the "correction decay" problem.

Research question: Design a study that would measure whether the belief-correction effects of fact-checks decay over time, and if so, how quickly. Your study design should: 1. Identify an appropriate false belief to test (one held by a substantial portion of the population). 2. Describe the experimental design, including treatment and control conditions. 3. Specify measurement intervals (when would you measure belief post-correction?). 4. Identify what outcomes you would measure (belief accuracy, confidence, behavior). 5. Anticipate at least three potential confounds and explain how you would address them.

Exercise 19.19 — Prebunking vs. Debunking Comparison

Prebunking (inoculation) and debunking (correction) represent two fundamentally different temporal strategies for addressing misinformation. Write a 600-word comparative analysis that:

1. Explains the theoretical mechanisms underlying each approach.
2. Summarizes the empirical evidence on the effectiveness of each approach.
3. Identifies populations or contexts in which each approach is likely to be more effective.
4. Addresses the practical implementation challenges of each approach.
5. Argues for a combined strategy that leverages the advantages of both.

Exercise 19.20 — Institutional Analysis: Funding and Independence

Fact-checking organizations must secure funding to operate, and their funding sources raise potential independence concerns. Research the funding sources of at least three major fact-checking organizations (options: PolitiFact, FactCheck.org, Full Fact, Africa Check, Snopes, The Washington Post Fact Checker). Then:

1. Identify each organization's major funding sources.
2. Assess whether any of the funding relationships pose potential conflicts of interest.
3. Evaluate how transparent each organization is about its funding.
4. Compare against the IFCN's transparency requirements.
5. Recommend best practices for fact-checking organization funding that would maximize both financial sustainability and editorial independence.

Part E: Extended Projects

Exercise 19.21 — Fact-Checking a Political Speech

Obtain the transcript of a political speech delivered within the past year (State of the Union addresses, convention speeches, major campaign speeches, and similar high-profile events are good choices). Conduct a systematic fact-check of the speech:

1. Identify all check-worthy factual claims in the speech (there may be 10-30 depending on the speech).
2. Select the five most significant claims for full fact-checking.
3. Conduct a verification of each claim, documenting your sources.
4. Write up your findings in the format of a published fact-check report, including ratings.
5. Reflect on the process: Which claims were easiest to verify? Which were most difficult, and why? What limitations did you encounter?

This project should produce a 1,500-2,000 word written fact-check report.

Exercise 19.22 — Building a Mini Fact-Check Database

Working in groups of 3-4, build a mini fact-check database by:

1. Collectively conducting 15-20 fact-checks on claims from a consistent domain (e.g., climate change claims, economic claims, health claims).
2. Recording each fact-check in a structured format: claim, date claimed, source, verdict, rating, evidence sources, date checked, checker.
3. Analyzing the resulting database: What patterns emerge? How many claims were true vs. false vs. mixed? Were any claims checked by multiple group members, and did you reach the same verdict?
4. Presenting your database and analysis to the class.

Exercise 19.23 — Simulated Fact-Checking Organization

In groups of 5-6, simulate the operation of a fact-checking organization for one week. Roles: editor, two fact-checkers, social media manager, and reader outreach coordinator. Tasks:

• Editor: selects three claims to check each day from a provided list of 20 candidate claims.
• Fact-checkers: each conducts two fact-checks per day.
• Social media manager: prepares tweets and posts for the fact-checks.
• Reader outreach coordinator: tracks reader responses and manages incoming claim submissions.

At the end of the simulation week, the group produces a written reflection on: 1. The difficulties encountered in claim selection. 2. The challenges in achieving consensus on ratings. 3. The time constraints experienced by fact-checkers. 4. What you would do differently with more resources.

Exercise 19.24 — Policy Proposal: Platform Integration

Write a 1,000-word policy proposal addressed to the CEO of a major social media platform (Facebook/Meta, Twitter/X, or YouTube) arguing for a specific approach to integrating fact-checking into the platform's content ecosystem. Your proposal should:

1. Identify the specific problem you are trying to solve.
2. Describe your proposed approach in operational detail.
3. Anticipate and address likely objections (cost, censorship concerns, partisan bias concerns, technical feasibility).
4. Propose metrics for evaluating the effectiveness of your approach.
5. Address potential unintended consequences.

Exercise 19.25 — Reflection: Personal Fact-Checking Practice

Maintain a log for one week in which you record every instance where you encountered a claim that you paused to verify or considered verifying. For each entry, record: 1. The claim and where you encountered it. 2. Whether you verified it and how. 3. What you found. 4. Whether you would have shared the claim before verifying, and whether your verification changed your decision.

At the end of the week, write a 500-word reflection on: - What patterns do you notice in what you paused to verify vs. what you accepted uncritically? - What does this reveal about your own information habits? - What changes to your information habits would make you a more effective fact-checker in your daily life?

Exercise 19.26 — Cross-Cultural Fact-Check Comparison

Select one factual claim that has been checked by fact-checking organizations in two or more different countries or language contexts (many claims related to COVID-19, climate change, immigration, and major international events were checked in multiple countries).

Compare: 1. Did organizations in different countries reach the same conclusion? 2. Did they draw on different evidence sources, and why? 3. Were there meaningful differences in how the claim was framed or what context was considered relevant? 4. What does this comparison suggest about whether fact-checking produces culturally universal or culturally variable verdicts?

Exercise 19.27 — Evaluating IFCN Certification

The IFCN's code of principles is the primary certification standard for professional fact-checking organizations. Read the full text of the IFCN code of principles on the Poynter Institute website. Then:

1. For each of the five principles, explain what it requires in operational terms.
2. Identify potential gaps or ambiguities in each principle.
3. Propose one specific amendment or addition to the principles that you believe would improve the quality or accountability of fact-checking organizations.
4. Research whether any IFCN-certified organization has had its certification questioned or revoked and analyze the circumstances.

Exercise 19.28 — Reading Nyhan & Reifler

Locate and read the original 2010 paper by Brendan Nyhan and Jason Reifler, "When Corrections Fail: The Persistence of Political Misperceptions" in Political Behavior. Then:

1. Summarize the paper's main findings in 200 words.
2. Evaluate the paper's methodology: What are its strengths? What are its limitations?
3. Research the subsequent scholarly debate about the "backfire effect": What have later studies found? Cite at least three studies published after 2015.
4. Write a 400-word essay on what the current scholarly consensus on corrections and belief change suggests for fact-checking practice.

End of Chapter 19 Exercises