Chapter 8 Key Takeaways: When AI Gets It Wrong
One-Page Summary
The Five Failure Types
| Type | Name | What Happens | Example |
|---|---|---|---|
| 1 | Wrong Answer | Standard error within the domain | Spam filter catches a real email |
| 2 | Hallucination | Confident fabrication, no factual grounding | LLM generates fake citations that look real |
| 3 | Distributional Shift | World differs from training data | Medical AI trained at one hospital fails at another |
| 4 | Adversarial | Deliberate input manipulation | Modified stop sign fools self-driving car |
| 5 | Cascading | Error propagates through connected systems | Transcription error changes diagnosis changes treatment |
Five Key Concepts
-
AI confidence and AI correctness are different things. Most AI systems are overconfident — their stated confidence exceeds their actual accuracy. A "92% confident" diagnosis may be wrong, and nothing in the output will tell you so.
-
Hallucinations are structural, not fixable. LLMs generate text by predicting probable next tokens, not by retrieving verified facts. This means they will always have the potential to produce plausible falsehoods. Mitigation techniques reduce but cannot eliminate this risk.
-
Distributional shift is invisible. AI systems cannot detect when they're operating outside their training distribution. They simply produce less reliable outputs without warning — the system doesn't know what it doesn't know.
-
Automation bias causes humans to over-trust machines, especially when outputs include confidence scores. This is driven by effort asymmetry (agreeing is easy), authority heuristics (the system seems expert), and accountability diffusion (blame is shared when you follow the machine).
-
Cascading failures amplify individual errors by propagating them through chains of interconnected systems, each treating the previous system's output as fact. Uncertainty is "laundered" — lost as data crosses system boundaries.
The VERIFY Framework
| Letter | Step | Key Action |
|---|---|---|
| V | Validate the source | Know which AI system produced the output and its limitations |
| E | Examine confidence | Treat confidence scores skeptically; check calibration |
| R | Reality-check | Does the output make sense? Is it too clean, too perfect? |
| I | Independently verify | Check key claims against independent sources, not the same AI |
| F | Flag the stakes | Match verification effort to consequences of error |
| Y | Yield to expertise | In high-stakes domains, defer to human experts |
Critical Vocabulary
- Hallucination: AI output that is fluent and confident but factually wrong
- Distributional shift: Mismatch between training data and deployment context
- Calibration: Alignment between stated confidence and actual accuracy
- Automation bias: Human tendency to over-rely on automated systems
- Cascading failure: Error that propagates through interconnected systems
- Graceful degradation: Failing safely by acknowledging uncertainty rather than producing confident wrong answers
- Adversarial input: Input deliberately designed to fool an AI system
Myth vs. Reality
Myth: AI systems will eventually stop making mistakes.
Reality: Error rates can be reduced but never eliminated. Certain failure modes — hallucination, distributional shift — are structural features of how these systems work, not bugs to be patched. The goal is not error-free AI but understanding how AI fails so you can use it wisely.
Project Checkpoint Reminder
For your AI Audit Report: Assess your system's vulnerability to each failure type, evaluate its confidence calibration, map cascading failure risks, and design a VERIFY-based verification plan for users.