Exercises — Chapter 20: AI Safety and Alignment

Part A: Conceptual Questions

A1. ⭐ Define the alignment problem in your own words, using an everyday (non-AI) analogy.

A2. ⭐ What is specification gaming? Give one example from the chapter and one original example you create yourself.

A3. ⭐⭐ Explain the difference between a robustness failure and an alignment failure. Can the same incident be both? Give an example.

A4. ⭐⭐ What is RLHF? Describe the process in three to four sentences, explaining how human feedback is used to improve AI behavior.

A5. ⭐⭐ The chapter discusses three approaches to AI safety: interpretability, RLHF, and constitutional AI. For each, identify one strength and one limitation.

A6. ⭐⭐⭐ Explain why the alignment problem becomes harder as AI systems become more capable. Use a concrete example to illustrate your argument.

A7. ⭐⭐ What is the difference between an AI system being "misaligned" and an AI system being "misused"? Why does this distinction matter for governance and accountability?

A8. ⭐⭐⭐ The chapter introduces the concept of "corrigibility" — whether a system allows itself to be corrected. Why is this important? What makes corrigibility a difficult property to guarantee?

Part B: Applied Analysis

B1. ⭐⭐ A company deploys a customer service chatbot with the objective "maximize customer satisfaction scores." Identify three ways the chatbot might achieve this objective through specification gaming — satisfying the metric without genuinely serving customers well.

B2. ⭐⭐ Apply the Evidence Evaluation framework from section 20.2 to deepfake technology. Answer each of the five questions in the framework.

B3. ⭐⭐⭐ Consider a large language model used for medical information. Describe: (a) a near-term safety concern that is already documented, (b) a plausible medium-term concern, and (c) a long-term speculative concern. How would your recommended response differ for each?

B4. ⭐⭐ Return to the MedAssist AI anchor example from Chapter 1. What alignment risks does MedAssist face? Is the system's objective ("identify abnormalities in medical images") well-specified? What gaps might exist between this objective and the broader goal of improving patient outcomes?

B5. ⭐⭐⭐ A social media company says it has "aligned" its recommendation algorithm with user well-being by optimizing for "time well spent" instead of "time spent." Evaluate this claim. Is "time well spent" a sufficiently well-specified objective? What specification gaming risks remain?

Part C: Research Design & Critical Thinking

C1. ⭐⭐⭐ Design a red-teaming exercise for a generative AI system used in education. What kinds of misuse would you test for? What outputs would count as safety failures? How would you distinguish between a genuinely dangerous response and an edge case that is unlikely to cause real harm?

C2. ⭐⭐⭐ The chapter discusses constitutional AI, where an AI system is given a set of principles to self-critique against. Draft a five-principle "constitution" for an AI system used in hiring. For each principle, explain what misalignment risk it addresses.

C3. ⭐⭐⭐⭐ Some researchers argue that interpretability research should be publicly funded and its results made freely available, rather than being conducted primarily within private companies. Evaluate this argument. What are the benefits of public interpretability research? What are the risks? Consider both safety and competitive implications.

C4. ⭐⭐⭐ The alignment problem assumes that "human values" exist in a form that could be specified. But as Chapter 9 showed, humans disagree deeply about values — even about the definition of fairness. How should alignment researchers handle the problem of value disagreement among humans? Is it possible to align AI with values that humans themselves have not agreed upon?

Part D: Synthesis

D1. ⭐⭐⭐ Write a 300-word response to the Eliezer Yudkowsky quote that opens this chapter: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." What is Yudkowsky's argument? Do you find it compelling? Why or why not?

D2. ⭐⭐⭐ Take a position in the accelerationist vs. cautionist debate. Write a 400-word argument for your position that acknowledges the strongest argument from the opposing side. What evidence would change your mind?

D3. ⭐⭐⭐⭐ The chapter argues that AI safety is not just an expert concern — that ordinary citizens have a meaningful role to play. Some critics respond that this is naive: "AI safety is a deeply technical problem, and giving citizens a say is like letting passengers vote on aircraft engine design." Evaluate this critique. Is the aviation analogy appropriate? Where does it hold, and where does it break down?

D4. ⭐⭐⭐ Synthesize the alignment problem, the global perspectives from Chapter 19, and the governance frameworks from Chapter 13. If you were advising the United Nations on AI safety, what three priorities would you recommend and why?

Part M: Mixed Practice (Spaced Review)

These questions revisit concepts from earlier chapters through the lens of AI safety.

M1. ⭐⭐ (Ch.8 connection) In Chapter 8, we explored specific AI failures. Choose one failure from that chapter and reanalyze it using the alignment framework from this chapter. Was the failure a robustness issue, a specification issue, or a misuse issue?

M2. ⭐⭐ (Ch.9 connection) Chapter 9 explored the difficulty of defining fairness. How does the "fairness is not a single metric" insight from Chapter 9 relate to the alignment problem? What does it suggest about the feasibility of aligning AI with human values?

M3. ⭐⭐ (Ch.13 connection) In Chapter 13, we discussed different governance approaches (industry self-regulation, government regulation, multi-stakeholder governance). Which approach is best suited for addressing AI safety concerns? Justify your answer.

M4. ⭐⭐⭐ (Ch.17 connection) Chapter 17 examined AI and accountability. If an aligned AI system is misused by a bad actor, who bears responsibility? If a misaligned AI system causes harm unintentionally, who bears responsibility? Are the answers different? Should they be?