Chapter 30 Further Reading: Verifying AI Output — Fact-Checking Workflows

Fact-Checking Methodology and Practice

1. Graves, L. (2016). Deciding What's True: The Rise of Political Fact-Checking in American Journalism. Columbia University Press. The foundational scholarly treatment of organized fact-checking as a journalistic practice. Though focused on political fact-checking organizations, the methodological principles — how to triage, how to calibrate confidence, how to handle uncertainty, how to communicate degrees of accuracy — apply broadly to professional verification practice. Not AI-specific, but deeply relevant.

2. International Fact-Checking Network (IFCN) — Code of Principles (poynter.org/ifcn) The IFCN's Code of Principles, which signatory fact-checking organizations commit to, articulates standards for non-partisan fact-checking methodology. Reading how professional fact-checkers define their practice provides a useful benchmark for individual professional verification practices. Free to access.

3. First Draft — Verification Handbook (firstdraftnews.org) First Draft has produced substantial free resources on verification methodology developed for journalists and researchers working with user-generated content and digital sources. The verification handbook covers source authentication, image and video verification, and claim verification at a level of practical detail that translates to professional AI output verification. Available free online.

Research on Verification Behavior and Failures

4. Pennycook, G., & Rand, D. G. (2019). Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences, 116(7), 2521-2526. Research showing that simple prompts to consider accuracy significantly improve people's ability to evaluate information quality. Directly relevant to the question of why built-in verification steps (rather than ambient awareness) are more effective — the structure of attention matters.

5. Swire-Thompson, B., & Lazer, D. (2020). Public health and online misinformation: challenges and recommendations. Annual Review of Public Health, 41, 433-451. A review article examining how false information spreads and persists. The mechanisms described — fluency of presentation as a credibility cue, repetition effects, difficulty of correction — are directly applicable to the AI hallucination problem and explain why "verify then trust" is a more protective standard than "trust but verify."

6. Amazeen, M. A. (2015). Revisiting the epistemology of fact-checking. Journalism Practice, 9(1), 23-36. An academic analysis of fact-checking epistemology that addresses the fundamental question of what verification establishes and what it cannot. The discussion of degrees of certainty and the limits of source-based verification is valuable for calibrating what a verification practice can and cannot guarantee.

Primary Source Databases and Verification Tools

7. CrossRef Member Reports and API Documentation (crossref.org/documentation) The full documentation for CrossRef, the DOI registration authority. Understanding how DOI registration works — what it guarantees and what it doesn't — provides a more sophisticated understanding of what a DOI lookup actually verifies. Free.

8. Semantic Scholar (semanticscholar.org) — API and Research Use Documentation Semantic Scholar's documentation explains its indexing methodology and what differentiates it from Google Scholar for verification purposes. Useful for understanding which database to use for which types of academic citation verification. Free.

9. USA.gov Guide to Government Databases (usa.gov/statistics) The US government's own guide to finding official data across federal agencies. Essential for understanding where authoritative government data lives across BLS, Census, BEA, CDC, and other agencies. Free.

AI-Specific Verification Challenges

10. Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On Faithfulness and Factuality in Abstractive Summarization. Proceedings of ACL 2020. A study showing that even when AI summarizes a provided source, it frequently introduces factual errors through the summarization process. This is directly relevant to the TVD framework — even source-provided content can be distorted in AI summaries, making verification against the original source worthwhile. Available free through ACL Anthology.

11. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W. T., Koh, P. W., ... & Hajishirzi, H. (2023). FActScoring: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Proceedings of EMNLP 2023. The research paper introducing FActScoring, a method for evaluating AI factual accuracy at the level of individual atomic claims. Understanding how researchers measure hallucination rates provides context for interpreting reported accuracy statistics and understanding what "verification" means in a technical sense. Available through ACL Anthology.

12. Rawlinson, K. (2023, May 27). ChatGPT invented a sexual harassment scandal and named a real law professor as the perpetrator. The Guardian. A journalistic account of a specific AI hallucination case involving false accusations against a named real person. Illustrates the severity of consequences in the most extreme hallucination category — false statements about individuals — and underscores why attribution verification is the highest-stakes verification category. Free to read.

Workflow and Process Design

13. Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18-19. Klein's "premortem" methodology — imagining the project has already failed and working backward to identify what went wrong — is directly applicable to building AI verification workflows. Before deploying AI-assisted content, asking "what would have to go wrong for this to cause a problem?" surfaces the verification requirements that matter most.

14. Gawande, A. (2009). The Checklist Manifesto: How to Get Things Right. Metropolitan Books. Gawande's case for systematic checklists in professional practice, drawn from aviation, surgery, and construction, provides the strongest argument for structural verification approaches over vigilance-based ones. The mechanisms he identifies — cognitive overload, the complexity problem, the failure of expertise alone to prevent known errors — map precisely onto the AI verification challenge.

15. Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A Flaw in Human Judgment. Little, Brown Spark. Kahneman and colleagues' analysis of "noise" — the variability in human judgment that occurs even when people are trying to be consistent. Relevant to why individual verification practices are inconsistent (noise is high when the task is ad hoc) and why systematic approaches reduce error rates. The chapter on "noise audits" provides a useful framework for assessing your current verification practice.

Ongoing Resources

Full Fact (fullfact.org/about/automated-fact-checking) — Full Fact publishes research on automated fact-checking tools and their limitations, useful for understanding the current state of the art in AI-assisted verification.
Duke Reporters' Lab (reporterslab.org/fact-checking) — maintains a database of global fact-checking organizations and tracks trends in the field.
The Markup (themarkup.org) — investigative journalism organization that publishes technically rigorous investigations of AI systems and data practices, including verification-relevant research.