Chapter 8 Quiz: When AI Gets It Wrong

Test your understanding of the concepts from this chapter. Try answering each question before checking the explanation.

Question 1. A language model asked to write a biography of a real scientist includes three fabricated research papers in its response. The papers have plausible-sounding titles, correct formatting, and real journal names — but they don't exist. This is an example of:

a) Distributional shift b) Cascading failure c) Hallucination d) Adversarial attack

Answer **c) Hallucination.** The language model is generating text that follows the statistical patterns of academic citations — correct format, plausible structure — without reference to whether the cited works actually exist. Hallucinations are characterized by fluent, confident outputs that are factually wrong. This isn't distributional shift (the model is within its training domain), adversarial attack (no one manipulated the input), or cascading failure (this is a single system error, not a chain of errors).

Question 2. A skin cancer detection AI was trained primarily on images of skin lesions on lighter skin tones. When used to evaluate patients with darker skin tones, its accuracy drops significantly. This is an example of:

a) Hallucination b) Distributional shift c) Cascading failure d) Automation bias

Answer **b) Distributional shift.** The model was trained on data with one distribution (predominantly lighter skin tones) and is being deployed in a context with a different distribution (darker skin tones). The features the model learned to associate with malignancy may present differently on different skin tones, causing the model's learned patterns to fail. This is also connected to the training data bias concepts from [Chapter 4](../../part-02-how-ai-works/chapter-04-data/index.md) — the training data didn't represent the full population.

Question 3. A radiology AI reports "95% confidence" in its diagnosis. Research shows that across many cases where this AI reports 95% confidence, it is actually correct only 78% of the time. This gap between stated confidence and actual accuracy indicates:

a) The system is well-calibrated b) The system is poorly calibrated (overconfident) c) The system is hallucinating d) The system has experienced distributional shift

Answer **b) The system is poorly calibrated (overconfident).** Calibration refers to the alignment between stated confidence and actual accuracy. A well-calibrated system would be correct about 95% of the time when it states 95% confidence. Being correct only 78% of the time when claiming 95% confidence means the system's confidence scores systematically overstate its actual accuracy. This is a common problem in deep learning models and one reason why confidence scores should be treated skeptically.

Question 4. A physician consistently accepts an AI diagnostic tool's recommendations without independent review, even in ambiguous cases where the physician's own training would suggest a different approach. This behavior pattern is best described as:

a) Hallucination b) Cascading failure c) Automation bias d) Distributional shift

Answer **c) Automation bias.** Automation bias is the human tendency to over-rely on automated systems, especially when those systems appear confident and authoritative. It's driven by effort asymmetry (agreeing is easier than disagreeing), the authority heuristic (the system seems sophisticated), alert fatigue (reviewing many AI outputs becomes tiring), and accountability diffusion (blame is shared if you followed the machine's recommendation). This is a human behavior problem, not an AI system error — but it amplifies the impact of AI errors.

Question 5. Which of the following best explains why large language models hallucinate?

a) They have bugs in their code that produce random errors b) They are designed to deceive users c) They generate text by predicting probable next tokens, not by retrieving verified facts d) Their training data contains only false information

Answer **c) They generate text by predicting probable next tokens, not by retrieving verified facts.** LLMs don't have a database of true statements to consult. They've learned statistical patterns of language from their training data — which words tend to follow which other words. When they generate a citation, they're producing a sequence of tokens that *looks like* a citation based on the millions of citations in their training data. The result is structurally plausible but not necessarily factually grounded. This is a fundamental architectural feature, not a bug or intentional deception.

Question 6. An AI-powered transcription system misinterprets a doctor's notes. The error propagates through three subsequent AI systems (clinical decision support, insurance pre-authorization, and treatment recommendation), each making decisions based on the incorrect transcription. This scenario illustrates:

a) Hallucination b) Distributional shift c) Automation bias d) Cascading failure

Answer **d) Cascading failure.** A cascading failure occurs when an error in one system propagates through interconnected systems, with each system treating the previous system's output as reliable input. In this scenario, the initial transcription error is amplified as each downstream system makes decisions based on corrupted data — and none of the downstream systems can detect the upstream error. This is exacerbated by "confidence laundering," where uncertain information loses its uncertainty markers as it passes through system boundaries.

Question 7. A researcher discovers that by changing just three pixels in a medical image, they can cause a diagnostic AI to change its classification from "benign" to "malignant" with high confidence. This is an example of:

a) Distributional shift b) An adversarial attack c) A hallucination d) Poor calibration

Answer **b) An adversarial attack.** Adversarial inputs are deliberately crafted to exploit vulnerabilities in AI systems — small, carefully designed perturbations that cause dramatic changes in the system's output. The key indicator is intentional manipulation: someone deliberately modified the input to fool the system. This is different from distributional shift (unintentional mismatch), hallucination (fabricated output without input manipulation), or poor calibration (systematic overconfidence).

Question 8. The chapter argues that hallucination in language models is "structural, not fixable." Which of the following statements best captures what this means?

a) No one is working on reducing hallucinations b) Hallucinations will always occur at the same rate regardless of improvements c) Because LLMs work by predicting probable text rather than retrieving verified facts, the potential for plausible falsehoods can be reduced but not eliminated d) Language models intentionally produce false outputs

Answer **c)** The claim that hallucination is "structural" means it arises from the fundamental architecture of how LLMs work — generating text through next-token prediction. Techniques like RAG, RLHF, and chain-of-thought prompting can significantly reduce hallucination rates, but they can't eliminate the possibility entirely because the underlying mechanism (producing statistically probable text) can always generate text that is plausible but false. This doesn't mean no improvement is possible (option b) — it means the problem can be managed, not solved.

Question 9. In the VERIFY framework, what does the "F" stand for, and why is it important?

a) "Find alternatives" — because you should always seek a second AI opinion b) "Flag the stakes" — because verification effort should match the consequences of an error c) "Filter the data" — because raw AI output needs preprocessing d) "Fix the model" — because errors indicate the model needs retraining

Answer **b) "Flag the stakes" — because verification effort should match the consequences of an error.** Not all AI outputs require the same level of verification. An AI recommending a movie requires minimal verification; an AI recommending a medical treatment requires extensive verification. The F in VERIFY reminds you to assess how much it matters if the output is wrong, and to calibrate your verification effort accordingly. This is a practical principle: exhaustive verification of every AI output is impossible, so directing your effort toward high-stakes outputs is both more efficient and more protective.

Question 10. The chapter's threshold concept states: "AI confidence and AI correctness are different things." A colleague argues: "If AI confidence doesn't indicate correctness, then confidence scores are useless — we should ignore them entirely." How would you respond?

a) Your colleague is right — confidence scores provide no useful information b) Confidence scores from well-calibrated systems are genuinely informative, but most systems are poorly calibrated, so they should be treated as rough guides rather than reliable indicators of correctness c) Confidence scores are always reliable — the chapter exaggerates the problem d) Confidence scores are only useful for recommendation systems, not for classification or prediction

Answer **b)** Confidence scores are not useless — in well-calibrated systems, they provide genuinely valuable information about uncertainty. The problem is that many deployed AI systems are poorly calibrated (overconfident), and most users don't know whether the system they're using is well-calibrated. The correct response is not to ignore confidence scores entirely but to treat them as rough guides, to investigate whether the system has been tested for calibration, and to always pair confidence scores with independent verification rather than treating them as reliable proxies for correctness. A well-calibrated 60% confidence is more useful than a poorly calibrated 95%.