Affiliate disclosure

Book titles on this page link to Amazon. As an Amazon Associate, DataField.Dev earns from qualifying purchases — at no additional cost to you.

Chapter 8 Further Reading: When AI Gets It Wrong

Meredith Broussard, Artificial Unintelligence: How Computers Misunderstand the World (MIT Press, 2018) Broussard, a data journalist and computer scientist, argues that AI systems fail in predictable ways because they're built on a flawed assumption she calls "technochauvinism" — the belief that technology is always the best solution. Her examples range from self-driving cars to standardized testing, and she grounds each in a clear understanding of how the underlying systems actually work. Particularly relevant to the distributional shift and overconfidence discussions in this chapter.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" (FAccT 2021) While technically a conference paper rather than a book, this landmark work is essential reading for understanding why LLMs hallucinate. The authors argue that language models learn to produce fluent text without understanding meaning, which makes hallucination a structural feature rather than a bug. The paper's central metaphor — the "stochastic parrot" — directly connects to Section 8.2's discussion of why LLMs fabricate content.

Gary Marcus and Ernest Davis, Rebooting AI: Building Artificial Intelligence We Can Trust (Pantheon, 2019) Marcus (a cognitive scientist) and Davis (a computer scientist) examine the gap between AI hype and AI reality. Their analysis of why current AI systems fail — brittleness, lack of common sense, inability to generalize — provides a technical complement to this chapter's more accessible treatment. The chapters on why deep learning alone is insufficient for reliable AI are particularly relevant.

Cade Metz, Genius Makers: The Mavericks Who Brought AI to Google, Facebook, and the World (Dutton, 2021) A narrative history of deep learning's rise that includes candid accounts of AI failures and near-misses from the people who built the systems. Metz is a technology reporter for the New York Times, and his reporting grounds the technical discussion in human stories. Useful context for understanding how the AI industry's culture and incentives contribute to the failure modes discussed in this chapter.

Benjamin Haibe-Kains et al., "Transparency and Reproducibility in Artificial Intelligence," Nature (2020) A commentary signed by over 30 researchers arguing that the lack of transparency in AI research — particularly in medical AI — makes it impossible to evaluate claimed performance or identify failure modes. Directly relevant to the distributional shift and calibration problems discussed in Sections 8.3 and 8.4.

Xiaoqian J. Jiang et al., "Opportunities and Challenges of AI in Healthcare," Nature Medicine (2021) A comprehensive review of AI in healthcare that documents the performance gap between AI systems tested on data from their training institutions and those tested on external data. The finding that most medical AI shows 10-15% accuracy drops on external data is cited in Section 8.3.

Benjamin Weiser, "Here's What Happens When Your Lawyer Uses ChatGPT," New York Times (May 27, 2023) The definitive journalistic account of the Schwartz/Avianca case discussed in Case Study 8.1. Weiser's reporting includes excerpts from court filings, the judge's sanctions order, and interviews with the lawyers involved. A primary source for understanding how hallucination manifests in professional settings.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger, "On Calibration of Modern Neural Networks," ICML 2017 The foundational study on why modern deep learning models are poorly calibrated — why their confidence scores don't reliably indicate accuracy. Technical but influential, and the basis for the overconfidence discussion in Section 8.4. Accessible to readers comfortable with basic statistics.

Janelle Shane, "The Danger of AI Is Weirder Than You Think" (TED Talk, 2019) Shane, author of the blog AI Weirdness, gives a 12-minute talk about how AI systems fail in unexpected ways. Her examples are humorous but instructive: AI generating bizarre recipes, creating absurd pickup lines, and failing at tasks that seem simple to humans. A light but substantive introduction to AI failure modes.

"AI Gone Wrong" podcast series, MIT Technology Review A series of episodes examining real-world AI failures across domains, from healthcare to criminal justice to autonomous vehicles. Each episode combines technical analysis with human-impact reporting. Particularly relevant episodes cover medical AI errors and algorithmic decision-making failures.

NIST AI Risk Management Framework (AI RMF 1.0) The U.S. National Institute of Standards and Technology's framework for identifying, assessing, and managing AI risks. While designed for organizations, the framework's taxonomy of AI risks is a useful reference for individual AI literacy. Freely available at nist.gov/artificial-intelligence.

"How to Read an AI Research Paper" — a guide Understanding AI failures often requires reading the technical literature. Several resources exist to help non-specialists read AI research papers critically. Look for guides from the Alan Turing Institute, the Partnership on AI, or university courses on AI literacy.

For Deep Dive Readers

Ji, Z., Lee, N., Frieske, R., et al., "Survey of Hallucination in Natural Language Generation," ACM Computing Surveys (2023) The most comprehensive academic survey of hallucination in language models. Covers definitions, causes, detection methods, and mitigation strategies. Technical but well-organized, with a useful taxonomy that extends the classification in Section 8.2.

Amodei, D., Olah, C., Steinhardt, J., et al., "Concrete Problems in AI Safety," arXiv:1606.06565 (2016) A foundational paper identifying five concrete problems in AI safety, including distributional shift, reward hacking, and scalable oversight. Written by researchers from Google Brain, Stanford, and Berkeley, it provides a rigorous framing of the failure modes discussed in this chapter within the broader AI safety research agenda.

Amershi, S., Begel, A., Bird, C., et al., "Software Engineering for Machine Learning: A Case Study," ICSE 2019 A study from Microsoft Research examining how AI failures differ from traditional software bugs. The paper documents how distributional shift, data drift, and cascading failures in ML pipelines require different engineering practices than conventional software. Relevant to the cascading failure discussion in Section 8.5.