Chapter 15 Exercises

Calibration: Why You Think You Know It When You Don't (and How to Fix It)

These exercises are designed to move beyond recognition toward genuine understanding and application. Resist the urge to flip back to the chapter while answering — the effort of retrieval is part of the learning process. And here's the meta-exercise: before answering each question, rate your confidence that you can answer it correctly (High / Medium / Low). After finishing, compare your confidence ratings to your actual performance. You'll be calibrating your calibration.

Part A: Conceptual Understanding

These questions test whether you can define and explain the chapter's core concepts. Aim for your own words, not quoted definitions.

A1. Define calibration in your own words. Then explain how calibration differs from resolution. Use a concrete example to show how a student could have good resolution but poor calibration.

A2. Explain the overconfidence effect. Why is it described as a "factory setting" rather than a character flaw? Name at least three cognitive cues that produce overconfidence and explain how each one works.

A3. What is the hard-easy effect? Describe the pattern: on what types of material are people overconfident, and on what types are they underconfident? Explain the cognitive mechanism that drives this pattern.

A4. Explain the unskilled-and-unaware problem in your own words. Why is it described as a "double bind"? What is the relationship between the skills needed to perform well and the skills needed to evaluate your own performance?

A5. What is hindsight bias, and how does it protect overconfidence from self-correction? What is foresight bias, and how does it inflate pre-test confidence? Name the antidote for each.

A6. Describe the three calibration training techniques presented in the chapter: structured prediction, calibration curve graphing, and confidence interval practice. For each, state the core procedure in one or two sentences.

A7. The chapter describes calibration unreliability as a threshold concept. What does that mean? Why is the realization that "your confidence is systematically biased" transformative rather than just informative?

A8. What is a calibration curve? What does it look like for a perfectly calibrated person? What does it typically look like for a real person?

Part B: Applied Analysis

These questions present scenarios and ask you to analyze them using the concepts from this chapter.

B1. Scenario: Priya is studying for her organic chemistry exam. She reviews all five chapters, does practice problems for each, and rates her confidence: "I'm about 85% ready." On the exam, she scores a 67%.

Analyze Priya's calibration error. What cognitive cues likely contributed to her inflated confidence? Was her error one of resolution (she couldn't tell which topics she knew) or calibration (her overall sense of readiness was off)? What should she do differently before the next exam?

B2. Scenario: Two students, Alex and Jordan, take the same psychology exam. Both score 72%. But their confidence patterns are different: - Alex predicted 90% on every question, regardless of difficulty. - Jordan predicted 90% on easy questions and 60% on hard questions.

Which student has better resolution? Which has better calibration? Which student is better positioned to improve their studying, and why?

B3. Scenario: Marcus is an experienced programmer who has been coding for ten years. He estimates that he can fix a particular bug in two hours. It takes him eight hours. He then estimates the next bug will take three hours. It takes seven.

Is Marcus displaying the overconfidence effect, the hard-easy effect, the unskilled-and-unaware problem, or some combination? Explain your reasoning. Why hasn't ten years of experience corrected his calibration?

B4. Scenario: After learning about overconfidence in this chapter, Lena decides to be cautious. For her next exam, she rates her confidence at 50% for every single question — even the ones she's sure about. She scores 78%.

Is Lena now well-calibrated? What's wrong with her approach? How does "strategic underconfidence" differ from genuine calibration improvement?

B5. Scenario: A medical student takes practice exams throughout the semester. For the first exam, her predicted score was 85% and her actual score was 72% (overconfident by 13 points). For the second, predicted 80%, actual 74% (overconfident by 6 points). For the third, predicted 78%, actual 76% (overconfident by 2 points). For the fourth, predicted 77%, actual 78% (underconfident by 1 point).

Describe what's happening to this student's calibration over time. What is producing this improvement? Is she getting better at medicine, better at self-assessment, or both?

B6. Scenario: After finishing an exam, Roberto says to his friend: "I definitely got question 14 right — it was about mitosis, and I studied that hard." When the grades come back, Roberto got question 14 wrong. He says: "Oh, right, I actually wasn't sure about that one. The wording was tricky."

Identify the metacognitive illusion at work. How is Roberto's memory of his pre-question confidence being retroactively edited? What technique could prevent this?

Part C: Real-World Application

These questions ask you to apply chapter concepts directly to your own life.

C1. Think about a recent exam, quiz, or assessment. Before taking it, how confident were you about your performance? After taking it (but before receiving your grade), how confident were you? When you got the grade, how did it compare to both predictions?

If your confidence exceeded your performance (overconfident), identify which cognitive cues — fluency, familiarity, availability, or coherence — likely inflated your prediction. If your confidence fell short of your performance (underconfident), speculate on why.

C2. Conduct a mini-calibration exercise right now. Pick a course you're currently taking. List 10 specific concepts or facts from that course. For each, rate your confidence (50%-100%) that you could define or explain it correctly if asked right now. Then, without looking anything up, try to define or explain each one. Compare your confidence ratings to your actual performance. What's your calibration gap?

C3. Start a calibration tracking log. For your next three quizzes or exams, record: (a) your predicted score before taking the assessment, (b) your felt score immediately after finishing, and (c) your actual score. Track the gap. Which prediction — the before or the after — is more accurate? What does this tell you about foresight bias?

C4. Try the confidence interval technique on a real-world prediction. For your next exam or assignment, instead of predicting a single score, give a range: "I'm 90% sure my score will fall between ___ and ___." After you get the result, check whether it fell within your range. If it didn't, your interval was too narrow — a sign of overconfidence.

C5. Identify a domain where you suspect you're well-calibrated (your confidence usually matches your performance) and one where you suspect you're poorly calibrated. What's different about these two domains? Do you get more frequent feedback in the well-calibrated domain? Do you have more experience? More honest practice?

Part D: Synthesis and Critical Thinking

These questions require you to integrate multiple concepts, evaluate arguments, or think beyond what the chapter explicitly stated.

D1. The chapter argues that overconfidence is a "factory setting" that persists despite experience. But some people are genuinely well-calibrated — forecasters, some expert diagnosticians, experienced poker players. What do these well-calibrated individuals have in common? What features of their training or environment produce better calibration? Use the chapter's framework to speculate.

D2. The unskilled-and-unaware problem suggests that the least knowledgeable people are the most overconfident about their knowledge. But the chapter also notes that experts can be overconfident about hard problems. Is expert overconfidence the same phenomenon as novice overconfidence, or are they different mechanisms? Make a case for each position.

D3. Consider the relationship between calibration and motivation. On one hand, overconfidence might help motivation by keeping you optimistic about your chances of success. On the other hand, accurate calibration helps you allocate effort where it's needed. Is there an optimal level of overconfidence — a sweet spot where you're slightly overconfident but not so much that you misallocate effort? Or is accurate calibration always better?

D4. The chapter presents hindsight bias and foresight bias as "protective mechanisms" for overconfidence. But these biases exist for evolutionary reasons — they may have served adaptive functions. Can you think of contexts where overconfidence is genuinely useful — where being poorly calibrated might lead to better outcomes than being well-calibrated? (Think about contexts like entrepreneurship, exploration, or social interaction.)

D5. Compare calibration training to physical training. A weight lifter who thinks they can bench press 250 pounds but can only lift 200 pounds is "overconfident." They could get hurt. But does the metaphor hold perfectly for cognitive calibration? Are there important differences between physical and cognitive overconfidence? Are the consequences symmetrical?

Part M: Mixed Practice — Retrieval from Earlier Chapters

These questions deliberately pull from earlier chapters to promote interleaving and spaced retrieval. Answer from memory.

M1. (From Chapter 8) What is a fluency illusion? How does the fluency illusion directly contribute to the overconfidence effect described in this chapter?

M2. (From Chapter 10) Explain the concept of desirable difficulties. How do desirable difficulties provide calibration data — that is, how does struggling with material help you assess your actual level of understanding?

M3. (From Chapter 13) What is the difference between a judgment of learning (JOL) and a feeling of knowing (FOK)? Which of these is more susceptible to calibration errors, and why?

M4. (From Chapter 8) The fluency illusion and the foresight bias are closely related. Describe the connection. How does rereading material (Chapter 8) create foresight bias (Chapter 15)?

M5. (Integration) Create a concept map or written description showing how these concepts connect: fluency illusions (Ch 8), desirable difficulties (Ch 10), delayed JOLs (Ch 13), overconfidence effect (Ch 15), and calibration training (Ch 15). Show how each concept feeds into or corrects the others.

Part E: Research and Extension (Optional)

These questions go beyond the chapter content. They're designed for students who want to explore further, or for use in research papers and advanced discussions.

E1. Research the Brier score — the quantitative measure of calibration mentioned in this chapter. How is it calculated? What makes it a better measure of calibration than simply calculating the average confidence-accuracy gap? What are its limitations?

E2. The Good Judgment Project, led by Philip Tetlock, trained volunteers to make calibrated predictions about geopolitical events. Some volunteers ("superforecasters") achieved remarkable calibration accuracy. Research the techniques used in this project. How do they compare to the calibration training techniques described in this chapter? What additional techniques did the superforecasters use?

E3. Research the relationship between expertise and calibration in a specific domain (medicine, law, finance, weather forecasting). In your chosen domain, how well-calibrated are experts? Does expertise improve calibration, and if so, by how much? Are there domains where experts are less well-calibrated than novices?

E4. Design a study to test whether calibration training in one domain (e.g., general knowledge) transfers to calibration in another domain (e.g., academic course material). Specify your hypothesis, participants, experimental design, measurements, and potential confounds.

E5. The chapter mentions that overconfidence may be socially rewarded. Research the concept of "confidence heuristic" — the tendency for people to trust confident speakers more than uncertain ones. How does this social dynamic interact with individual calibration? Does social pressure for confidence make individual calibration harder to maintain?

End of Chapter 15 Exercises. Complete these before starting Chapter 16 to maximize the spacing effect on your retention of this chapter's material.