Capstone 2 Student Guide: The Misinformation Tracker

Overview

You are joining OpenDemocracy Analytics (ODA) as a research associate working with Sam Harding (lead data journalist) and Adaeze Nwosu (executive director) to build and operate ODA's first public-facing misinformation tracker for the Garza-Whitfield Senate race.

This capstone integrates everything you have learned about the political media ecosystem (Chapters 23-24), misinformation research (Chapter 26), text analysis (Chapters 27 and 37), data collection systems (Chapter 3), survey design (Chapter 7), and equity in data (Chapter 39). The tracker you build represents ODA's public commitment to political accountability.

This is an applied, judgment-intensive project. There are no perfectly right answers — only well-reasoned ones. Your grade reflects the rigor of your process and the quality of your reasoning, not whether your ratings match any external answer key.

Learning Objectives

By the end of this capstone, you will be able to:

1. Apply the Wardle-Derakhshan typology and ODA's four-category rating system to specific political claims
2. Design and document a systematic claim intake pipeline that distinguishes verifiable factual claims from rhetoric
3. Build an automated text analysis pipeline (using VADER and TF-IDF classification) for preliminary claim detection
4. Conduct a spread analysis for a specific false claim, including correction gap documentation
5. Design a public-facing accountability dashboard that communicates complex judgments to a general audience
6. Apply ethics and equity frameworks to evaluate a journalistic accountability system
7. Write a claim rating in the format appropriate for public publication

Required Readings and Resources

Before beginning the capstone deliverables, review:

• Chapter 43 (this capstone's main text) — complete
• Chapter 26 (Misinformation and Fact-Checking) — especially Sections 2, 4, and 6
• Chapter 23 (Media Ecosystem Analysis) — especially Sections 3 and 5
• Chapter 27 (Analyzing Political Text) — especially Sections 2-4
• Chapter 39 (Race and Data Justice) — especially Sections 4 and 5
• Wardle, C. and Derakhshan, H. (2017), Information Disorder — available in Appendix D
• ODA data documentation (this student guide's Data Appendix section)

Optional additional resources for deeper engagement: - PolitiFact's methodology page (real-world fact-checker process comparison) - Washington Post Fact Checker methodology - First Draft's verification handbook

Deliverables

This capstone has five deliverables. Deliverable weights are specified in the rubric.

Deliverable 1: Claim Taxonomy and Intake Design (15% of grade)

What you submit: A written document (1,500-2,000 words) that specifies:

1. Your claim taxonomy: Define the four-step decision tree for claim entry (Is it verifiable? Is it material? Is it attributable? Is it already adjudicated?). Give one example claim that clears all four steps and one that does not, with brief explanation of why.

2. Your rating rubric: Operationalize the A1-A4 accuracy scale and I1-I3 impact scale with concrete decision criteria. For each rating category, specify what evidence would be sufficient to assign that rating and what would distinguish it from adjacent categories.

3. Nonpartisanship protocol: Describe at least three specific procedures you would implement to ensure the tracker applies the same standards to both campaigns. Be specific — "we will be balanced" is not a procedure.

4. Limitations statement: Write the limitations statement that would appear on your tracker's public page (200-300 words). Be honest about what the tracker cannot do.

What good looks like: Precise definitions with clear decision rules, examples that are specific rather than generic, limitations that are genuinely limiting rather than merely formal.

Deliverable 2: Spread Analysis for Two Claims (25% of grade)

What you submit: Two complete spread analyses, each following the format used in Section 4 of the main capstone text.

Using the oda_media.csv dataset as your primary data source, select two claims from the following list (one must be from the Garza side and one from the Whitfield side) and conduct a complete spread analysis for each:

Available claims for analysis: - W-1: "Maria Garza supports open borders." - W-2: "Garza's healthcare plan would eliminate private insurance." - W-3: "Violent crime rose 40 percent under Garza's tenure as AG." - G-1: "Tom Whitfield outsourced jobs to China." - G-2: "Whitfield's plan would cut Medicare by $200 billion."

For each claim, your analysis must address:

A. Claim documentation: Exact claim text, original source, date, and source type (paid advertising, candidate statement, PAC communication, social media, etc.)

B. Verification analysis: What primary sources are relevant? What do they show? What is your A1-A4 rating and why? What would change your rating?

C. Spread pathway: Trace the claim's likely pathway through the media ecosystem using the five-stage model (Origin → Partisan Amplification → Mainstream Pickup → Fact-Checker Response → Background Radiation). Use data from oda_media.csv to support your characterization.

D. Correction gap estimate: What is the estimated reach of the original claim vs. the reach of corrections? Use oda_media.csv to identify any fact-checker coverage of your selected claim, and estimate relative audience sizes using source and source_type metadata.

E. Impact rating: Assign an I1-I3 rating with justification. Specify what data you used to estimate impact.

Format: Each analysis should be approximately 800-1,200 words.

Deliverable 3: Automated Detection Pipeline (25% of grade)

What you submit: A Python notebook or script that implements Sam's automated detection pipeline, applied to the oda_media.csv dataset. The pipeline must include:

Required components:

1. VADER sentiment flagging: Flag articles with compound sentiment score above |0.6| combined with presence of at least one watchlist term. Report: how many articles are flagged? What is the precision of this flag compared to articles with fact-checker ratings in the dataset?

2. TF-IDF classifier: Train a binary classifier (misinformation vs. accurate) on labeled examples from oda_media.csv using the factcheck_rating column. Report: 5-fold cross-validation F1 score, precision, and recall. List the top 10 most predictive features for each class.

3. False positive analysis: From the flagged articles that your classifier assigns high probability to, sample 20 and manually assess whether they are true positives (actual misinformation) or false positives (accurate content, or meta-coverage of misinformation). Report your precision estimate from this manual sample.

4. Source credibility downweighting: Implement the source credibility adjustment (Section 5) using source_type metadata from oda_media.csv. Show how this changes the flag rate for content from newspaper vs. digital vs. social media sources.

Optional challenge component (for extra credit): Implement a topic model (LDA) with K=8-12 topics on the oda_media.csv text corpus, and identify which topics most closely correspond to the five tracked claims in Section 4. Report your methodology and findings.

Format: Submitted as a Jupyter notebook with markdown commentary explaining each step. Code must be reproducible — include all imports, assume only standard Python libraries (pandas, sklearn, numpy, matplotlib, vaderSentiment).

Deliverable 4: Draft Public Rating (20% of grade)

What you submit: A complete public-facing claim rating in ODA's format for one claim of your choosing (can be one of the claims from Deliverable 2 or a different claim from the same list).

The rating must include:

A. Rating card (what appears as the summary on the dashboard): Claim text, attribution, accuracy rating (A1-A4), impact rating (I1-I3), plain-language explanation in 150 words or fewer, and date.

B. Full methodology details (what appears in the "Methodology" tab): Complete evidence chain, all sources consulted (with citation), expert consultations (you may use fictional but realistic expert names and affiliations), campaign response (you may use a fictional but realistic response), and a clear statement of what would change the rating.

C. Spread data section: Claim reach estimate (original), correction reach (if applicable), and a brief correction gap statement.

D. Reviewer note: Write a one-paragraph internal note (not for public display) explaining the most difficult judgment call in this rating and how you resolved it.

Format: Approximately 1,000-1,500 words total across all four components.

Deliverable 5: Ethics and Equity Audit Report (15% of grade)

What you submit: A structured audit report (1,000-1,500 words) applying the frameworks from Section 7 and Section 8 to your tracker design.

The report must address:

A. Accountability structure: Who oversees your tracker? Describe at least two accountability mechanisms beyond "the lead analyst checks their own work."

B. Asymmetric amplification assessment: For one of the claims in your tracker, assess the amplification risk: Is this claim obscure enough that a prominent fact-check could bring it to new audiences? How does your tracker's minimum-reach threshold policy apply here?

C. Free speech boundary: Identify one example from the Garza-Whitfield race of content that might look like a misinformation claim but should not be rated (because it's opinion, prediction, values claim, or ambiguous expertise). Explain why it falls outside the tracker's scope.

D. Equity audit: Apply ODA's equity checklist (Section 8) to your tracker design. Identify at least two specific equity gaps in the approach described in the main capstone text, and propose a concrete remediation for each.

E. Campaign pushback scenario: Write a response to the following scenario: The Whitfield campaign sends an email accusing ODA of "conducting a partisan hit job" and demanding retraction of your Claim W-3 rating. The email includes one new piece of evidence: a quote from a criminologist who says crime statistics in this state are "genuinely complex and subject to interpretation." How do you respond?

Timeline and Suggested Work Schedule

This capstone is designed for 15-20 hours of work. A suggested schedule for a student working over two weeks:

Days 1-2 (3-4 hours): Read main capstone text; review prerequisite chapters listed above; download and explore oda_media.csv dataset

Days 3-5 (3-4 hours): Complete Deliverable 1 (Claim Taxonomy and Intake Design) — this establishes your framework for all subsequent work

Days 6-9 (4-5 hours): Complete Deliverable 2 (Spread Analyses) and Deliverable 3 (Python pipeline) — these can be done in parallel

Days 10-12 (3-4 hours): Complete Deliverable 4 (Draft Public Rating) — builds on your Deliverable 2 work

Days 13-14 (2-3 hours): Complete Deliverable 5 (Ethics and Equity Audit) and final review

Important: Do not leave the Python deliverable (Deliverable 3) for the final days. Debugging code takes time. Start it early.

Working with the ODA Dataset

The oda_media.csv dataset contains approximately 15,000 rows representing media coverage. Key columns for this capstone:

Data cleaning notes: - factcheck_rating contains NaN for most articles (the majority have not been fact-checked) - For classifier training, use only rows where factcheck_rating is not NaN - headline and excerpt may contain missing values — use .fillna('') before text processing

Loading the dataset:

import pandas as pd
df = pd.read_csv('oda_media.csv')
# Filter to labeled examples for classifier training
labeled = df[df['factcheck_rating'].notna()].copy()
# Create binary label
labeled['mislabel'] = labeled['factcheck_rating'].apply(
    lambda x: 1 if str(x).lower() in
    ['false', 'mostly false', 'misleading',
     'four pinocchios', 'three pinocchios'] else 0
)

Common Mistakes to Avoid

On claim rating: - Do not rate opinions, predictions, or arguments as misinformation — only verifiable factual claims - Do not assign A1 (Documented False) without citing specific primary source evidence that directly contradicts the claim - Do not confuse "I disagree with this position" with "this claim is false" - Do not apply different evidence thresholds to claims from different campaigns

On the Python pipeline: - Do not treat classifier output as ground truth — it is a screening tool, not a rating system - Do not use the full dataset (including unlabeled rows) as training data — only labeled examples have ground truth - Do not report only accuracy — for imbalanced classes, accuracy is misleading. Report precision, recall, and F1 for the misinformation class

On the ethics and equity audit: - Do not conflate "legal" with "ethical" — the ethics review asks whether something is right, not just whether it's permitted - Do not treat the equity checklist as a formality — identify real gaps and propose real remediations - Do not respond to the campaign pushback scenario with "we'll reconsider our rating" unless you can specify what new evidence would actually change the rating

Grading Criteria Summary

Detailed criteria appear in the Grading Rubric. At a high level, your work will be evaluated on:

1. Accuracy and rigor: Is your claim analysis grounded in evidence? Are your methodological choices defensible?
2. Nuance and judgment: Do you demonstrate the ability to distinguish between degrees of misinformation, rather than treating everything as binary?
3. Process transparency: Can a reader follow your reasoning from evidence to conclusion?
4. Technical correctness: Does your Python pipeline run correctly? Are the results interpreted accurately?
5. Ethics integration: Is the ethics and equity review substantive, or merely formal?

The most common reason students lose points on this capstone is not errors of fact — it is failures of rigor in their reasoning chain. Make your reasoning visible.

A Note on Difficulty

Sam Harding, when briefing ODA's advisory panel on the tracker's methodology, said: "The hardest part of this work is not the Python. The hardest part is being honest with yourself about what the evidence actually shows, especially when it's inconvenient."

That difficulty is intentional. This capstone is designed to put you in situations where you have to make hard judgment calls and document your reasoning. There is no answer key for "is this A2 or A3?" There is only the quality of the argument you make and the evidence you bring to bear.

Make the argument carefully. Show your work. And if you're genuinely uncertain, say so — epistemic honesty is a methodological virtue, not a weakness.

Good luck.