Chapter 10 Exercises

How to use these exercises: Work through the parts in order. Part A builds recognition skills, Part B develops analysis, Part C applies concepts to your own domain, Part D requires synthesis across multiple ideas, Part E stretches into advanced territory, and Part M provides interleaved practice that mixes skills from all levels.

For self-study, aim to complete at least Parts A and B. For a course, your instructor will assign specific sections. For the Deep Dive path, do everything.

Part A: Pattern Recognition

These exercises develop the fundamental skill of recognizing Bayesian reasoning -- and its absence -- across domains.

A1. For each of the following scenarios, identify the prior, the evidence, and the posterior. Then state whether the person in the scenario is reasoning in a Bayesian manner or committing base rate neglect.

a) A hiring manager interviews a candidate who gives a brilliant presentation. She immediately concludes that the candidate is highly qualified, without considering that 80 percent of the candidates who reach the interview stage give excellent presentations.

b) A doctor in a rural clinic considers the possibility that a patient's headache is caused by a brain tumor. He reflects that brain tumors are extremely rare (about 1 in 50,000 per year) and that headaches are common, so he orders standard tests before considering advanced imaging.

c) A security officer at an airport sees a passenger sweating and fidgeting. He concludes the passenger is probably a security threat, despite the fact that millions of innocent passengers pass through the airport each year and many of them sweat and fidget due to flight anxiety.

d) A teacher notices that a student's essay contains several phrases that sound unusually polished. Before accusing the student of using AI, she considers that this student has consistently demonstrated strong writing in class, making polished phrasing less surprising.

e) A financial analyst sees that a small biotech company's stock rose 15 percent in one day. He concludes that the company must have received positive drug trial results, without considering that small biotech stocks frequently experience large daily swings for no fundamental reason.

A2. The chapter describes how Bayesian reasoning has been rediscovered independently in multiple fields. For each of the following, identify the Bayesian structure (what serves as the prior, evidence, likelihood, and posterior?):

a) A chess player evaluating whether their opponent will play aggressively or defensively, based on the opponent's first few moves and the player's knowledge of the opponent's historical style.

b) A climate scientist updating projections of future warming based on a new study about cloud feedback mechanisms, combined with prior models.

c) A venture capitalist meeting a startup founder for the third time, updating her assessment of the founder's competence based on how well the founder addressed concerns from the second meeting.

d) A bird listening for the call of a predator amid forest sounds, with its alarm threshold calibrated by recent predator activity in the area.

e) A machine learning model being fine-tuned on new data after pre-training on a large general corpus.

A3. Classify each of the following as the prosecutor's fallacy, base rate neglect, or correct Bayesian reasoning. Explain your classification.

a) "The DNA match probability is one in a billion, therefore there is a one-in-a-billion chance the defendant is innocent."

b) "This screening test is 95 percent accurate, but the condition affects only 1 in 10,000 people, so a positive result is probably a false positive."

c) "Three witnesses independently identified the defendant, so even though eyewitness identification is unreliable, the convergence of three identifications substantially increases the posterior probability of a correct identification."

d) "The lie detector says he is lying, and lie detectors are 90 percent accurate, so there is a 90 percent chance he is lying."

e) "Given that this patient has both a family history of heart disease and elevated cholesterol (two pieces of evidence), and given the base rate of heart disease in his age group, the posterior probability of coronary artery disease is approximately 35 percent."

A4. The chapter argues that the immune system performs Bayesian updating. Identify analogous Bayesian processes in each of the following biological systems:

a) A plant adjusting its growth direction in response to light (phototropism).

b) Ant colonies adjusting foraging trails based on which paths yield food.

c) The human visual system interpreting ambiguous images (e.g., the Necker cube, optical illusions).

d) A dog learning which behaviors get rewarded by its owner.

A5. For each of the following pairs, explain which person has the more informative prior and why that matters for their posterior belief:

a) A first-year medical student vs. an experienced cardiologist interpreting an electrocardiogram.

b) A tourist vs. a longtime resident assessing whether a neighborhood is safe to walk through at night.

c) A novice wine taster vs. a sommelier tasting a wine and estimating its price.

d) A new employee vs. the CEO assessing whether a proposed strategy will work.

Part B: Analysis

These exercises require deeper analysis of Bayesian reasoning concepts.

B1. The False Positive Paradox in Different Domains. Consider the following three scenarios, all involving tests with 99 percent sensitivity and 99 percent specificity, but different base rates.

a) A drug test administered to all employees at a company where 5 percent use drugs. What is the positive predictive value (the probability that a positive test indicates actual drug use)? Show your reasoning using natural frequencies.

b) A terrorism screening system applied to all airline passengers, where the base rate of actual terrorists is approximately 1 in 10 million. What is the positive predictive value? What does this imply about the usefulness of the screening system?

c) A diagnostic test for a genetic condition applied to newborns in a population where the condition affects 1 in 500 babies. What is the positive predictive value?

d) Compare your three answers. What general principle emerges about the relationship between base rate and the usefulness of even highly accurate tests?

B2. Priors in Scientific Research. The chapter discusses the reproducibility crisis as a Bayesian problem.

a) Explain why a field that tests mostly false hypotheses (low prior probability of any individual hypothesis being true) should expect a high rate of false positives, even with standard statistical practices.

b) How does publication bias (the tendency to publish only significant results) worsen the problem from a Bayesian perspective?

c) Imagine two research fields: one where hypotheses are typically grounded in strong theoretical predictions (physics), and one where hypotheses are generated by exploratory data analysis (some areas of social psychology). How would the prior probability of hypotheses being true differ between these fields, and what would this imply for the expected rate of false positives?

d) Some researchers have proposed replacing p-values with Bayes factors (the ratio of the likelihood of the data under the alternative hypothesis to the likelihood under the null). What advantage would this have from a Bayesian perspective?

B3. The Prosecutor's Fallacy in Practice. Consider the following courtroom scenario:

A crime has been committed in a city of 2 million people. A forensic technique (not DNA, but a less precise method such as bite mark analysis) matches the defendant to the crime scene evidence. The forensic expert testifies that the probability of a random match is 1 in 10,000.

a) If the defendant was identified solely through a database search (no other evidence linking them to the crime), what is the approximate probability that the defendant is actually guilty, assuming any person in the city could have committed the crime? Show your Bayesian reasoning.

b) Now suppose, in addition to the forensic match, three independent eyewitnesses identified the defendant from a lineup. How does this additional evidence change the posterior probability of guilt?

c) Explain why the order in which evidence is presented should not matter in a Bayesian analysis but often does matter psychologically (anchoring effect).

d) A defense attorney argues: "The forensic technique has a known error rate of 5 percent -- meaning 5 percent of matches are false. Therefore there is a 5 percent chance my client is innocent." Is this the prosecutor's fallacy in reverse (the defense attorney's fallacy)? Explain.

B4. Bayesian Reasoning and the Explore/Exploit Tradeoff. The chapter mentions Thompson sampling as a Bayesian algorithm for the multi-armed bandit problem (Chapter 8).

a) Explain how a Bayesian agent's uncertainty about the value of an unexplored option makes exploration rational, even when current exploitation is producing good results.

b) How does accumulating evidence (through exploration) change the Bayesian agent's posterior about each option? When does this naturally lead to a shift from exploration to exploitation?

c) Compare Bayesian updating to the run-and-tumble strategy of E. coli (Chapter 8). In what sense is the bacterium doing something Bayesian? In what sense is it not?

B5. When Priors Go Wrong. The chapter argues that priors are not bias. But priors can be wrong. Analyze:

a) Give an example of a domain where a strong prior based on extensive experience is genuinely helpful and leads to better reasoning than a weak or absent prior.

b) Give an example of a domain where a strong prior based on cultural assumptions or stereotypes leads to worse reasoning than a weaker prior.

c) What distinguishes a helpful prior from a harmful one? Is the distinction about the source of the prior, the strength of the prior, or the willingness to update?

d) How does the concept of "calibration" (having beliefs that match actual frequencies) relate to the quality of priors?

Part C: Application

These exercises ask you to apply Bayesian reasoning to your own experience and context.

C1. Identify three beliefs you currently hold with moderate confidence (roughly 60-80 percent credence). For each, describe:

a) What evidence originally formed this belief (what is the basis for your prior?)

b) What new evidence would increase your confidence to above 90 percent?

c) What new evidence would decrease your confidence to below 40 percent?

d) Have you actively sought out evidence that might disconfirm this belief, or have you only encountered confirming evidence?

C2. Think about a decision you made recently that was based on a judgment about probability (e.g., "It will probably rain," "This candidate will probably succeed," "This investment will probably pay off").

a) What was your implicit prior? Where did it come from?

b) What evidence did you consider?

c) Did you give appropriate weight to the base rate, or did you focus primarily on the specific evidence at hand?

d) With the benefit of hindsight, was your posterior well-calibrated?

C3. Choose a domain you know well (your profession, a hobby, an area of expertise). Identify an instance where base rate neglect commonly occurs in that domain. Describe:

a) The base rate that is typically neglected.

b) The specific evidence that draws attention away from the base rate.

c) The resulting error (overconfidence in a rare event, or underconfidence in a common one).

d) How the error could be corrected by explicit Bayesian reasoning.

C4. The chapter describes how medical education could be improved by teaching physicians to think in natural frequencies. Design a brief training exercise for professionals in your field that would help them avoid base rate neglect in their domain.

Part D: Synthesis

These exercises require integrating Bayesian reasoning with concepts from multiple chapters.

D1. Bayesian Reasoning and Signal Detection (Chapter 6). The mammogram example appears in both Chapter 6 (as a signal detection problem) and Chapter 10 (as a Bayesian reasoning problem).

a) Map the signal detection framework (signal, noise, sensitivity, specificity, false positive, false negative) onto the Bayesian framework (prior, likelihood, posterior, base rate). Show that they are describing the same underlying structure.

b) Explain the ROC curve (Chapter 6) in Bayesian terms. What does moving along the ROC curve correspond to in terms of changing the prior or the decision threshold?

c) In what sense does Bayesian reasoning extend signal detection theory by adding something that signal detection theory alone does not include? (Hint: what role does the prior play?)

D2. Bayesian Reasoning and Power Laws (Chapter 4). The chapter discusses the reproducibility crisis as a Bayesian problem involving low prior probabilities.

a) In fields where the distribution of effect sizes follows a power law (most effects are small, a few are large), how should this affect the prior probability assigned to hypotheses about large effects?

b) How does the power-law distribution of research impact (most papers are rarely cited, a few are cited thousands of times) interact with the Bayesian analysis of the reproducibility crisis?

c) Argue that the "publish or perish" culture in academia creates incentives that systematically distort Bayesian reasoning. What priors get inflated? What evidence gets suppressed?

D3. Bayesian Reasoning and Feedback Loops (Chapter 2). Consider a feedback loop in which Bayesian updating is embedded:

a) A physician treats patients in a clinic. Her diagnostic beliefs (priors) affect which tests she orders. The test results (evidence) update her beliefs (posteriors). Her updated beliefs affect her next round of test ordering. Describe how this feedback loop could lead to either well-calibrated beliefs or systematically biased ones, depending on the structure of the feedback.

b) How does confirmation bias create a positive feedback loop that distorts Bayesian updating? Use the framework from Chapter 2 to analyze this.

c) Is there a negative feedback loop that could counteract confirmation bias? If so, what would it look like?

D4. Bayesian Reasoning and Emergence (Chapter 3). The chapter describes the immune system as a Bayesian updater. Consider the population of B cells as a system:

a) How does the collective behavior of the B cell population (emergence) relate to the individual Bayesian updating performed by each cell?

b) In what sense is the immune system's "belief" about a pathogen an emergent property that is not held by any individual cell?

c) Draw a parallel between the immune system's Bayesian search and the ant colony's foraging search (Chapter 3). What serves as the prior, evidence, and posterior in each case?

Part E: Extension

These exercises push beyond the chapter's content into more advanced territory.

E1. The Problem of the Prior. One of the deepest criticisms of Bayesian reasoning is the choice of prior. Investigate:

a) What is an "uninformative prior" or "objective prior"? Why have some Bayesians proposed using them, and what problems do they solve?

b) What is the problem with uninformative priors in high-dimensional spaces? (Hint: consider Bertrand's paradox or the reference prior problem.)

c) The chapter argues that frequentist methods have hidden priors. Give three specific examples of implicit prior assumptions in standard frequentist methods (e.g., choice of null hypothesis, choice of significance level, assumption of normality).

E2. Bayesian Reasoning and Artificial Intelligence. Modern AI systems use Bayesian reasoning in various ways.

a) Explain how a Bayesian neural network differs from a standard neural network. What does it estimate that a standard network does not?

b) How does the concept of a "prior" in Bayesian deep learning relate to regularization techniques (which penalize complex models)? Why is this connection important?

c) Large language models are sometimes described as performing "implicit Bayesian inference" over text. In what sense is this true, and in what sense is it misleading?

E3. Bayes' Theorem and Decision Theory. Bayes' theorem tells you what to believe. Decision theory tells you what to do.

a) Explain the distinction between belief updating (Bayesian inference) and decision-making (maximizing expected utility). Why are they related but not identical?

b) A doctor must decide whether to recommend a biopsy based on a positive mammogram. The Bayesian analysis says the probability of cancer is 9 percent. Should she recommend the biopsy? What additional information does she need (beyond the posterior probability) to make this decision?

c) How does the asymmetry of consequences (missing a cancer vs. an unnecessary biopsy) affect the optimal decision threshold? Is this a Bayesian consideration or a decision-theoretic one?

Part M: Mixed Practice (Interleaved Review)

These problems deliberately mix concepts from Chapters 6, 8, and 10 to strengthen retrieval and transfer.

M1. A new diagnostic test for a rare disease is being evaluated. The test has 95 percent sensitivity and 90 percent specificity. The disease affects 1 in 1,000 people.

a) What is the positive predictive value? (Use the Bayesian framework from Chapter 10.)

b) Draw this scenario in the signal detection framework from Chapter 6, identifying the signal, noise, hits, misses, false alarms, and correct rejections.

c) If the disease were more common (1 in 100 instead of 1 in 1,000), how would the positive predictive value change? How would this shift appear on an ROC curve?

M2. A search engine wants to show users the most relevant results. It starts with a prior estimate of relevance for each page based on page authority (PageRank). Then it updates based on the user's specific query (the evidence).

a) Frame this as a Bayesian problem. What is the prior? The evidence? The likelihood? The posterior?

b) Frame this as an explore/exploit problem (Chapter 8). When should the search engine explore (show less-established pages to test relevance) vs. exploit (show the pages it already knows are most relevant)?

c) How do these two framings complement each other?

M3. A military intelligence analyst receives a report suggesting that an adversary is preparing for an attack. The report comes from a source of uncertain reliability.

a) What is the analyst's prior (before the report)? What factors should shape this prior?

b) How should the reliability of the source affect the likelihood term in the Bayesian update?

c) Connect this to the signal detection problem from Chapter 6: what are the costs of false positives (raising an alarm when no attack is coming) vs. false negatives (failing to detect a real attack)?

d) How does this scenario relate to the explore/exploit tradeoff (Chapter 8)? Should the analyst "explore" (gather more intelligence) or "exploit" (act on the current information)?

M4. A scientist has conducted an experiment and obtained a p-value of 0.03 (below the conventional 0.05 threshold).

a) Explain why a p-value of 0.03 does not mean there is a 97 percent probability that the hypothesis is true. What does it mean?

b) Using the Bayesian framework, explain what additional information you would need to determine the probability that the hypothesis is true.

c) If the hypothesis was generated by data dredging (testing many hypotheses until one was significant), how does this affect the Bayesian analysis? Connect this to the signal/noise problem from Chapter 6.

d) What explore/exploit dynamic (Chapter 8) does data dredging represent? Is it exploration, exploitation, or something else?