Chapter 35 Exercises

How to use these exercises: Work through the parts in order. Part A builds recognition skills, Part B develops analysis, Part C applies concepts to your own domain, Part D requires synthesis across multiple ideas, Part E stretches into advanced territory, and Part M provides interleaved practice that mixes skills from all levels.

For self-study, aim to complete at least Parts A and B. For a course, your instructor will assign specific sections. For the Deep Dive path, do everything.

Part A: Pattern Recognition

These exercises develop the fundamental skill of recognizing the streetlight effect across domains.

A1. For each of the following scenarios, identify the "streetlight" (where the search is happening), the "park" (where the answer likely is), and the structural force keeping the search under the light (measurement availability, institutional incentives, or path dependence).

a) A school district evaluates teacher effectiveness using standardized test scores because they are the only data collected systematically across all schools.

b) A tech company measures user engagement by time-on-platform because click data is automatically recorded, even though user satisfaction may be declining.

c) A historian writes a history of a medieval kingdom based almost entirely on court records, because those are the documents that survived.

d) A public health agency tracks disease prevalence using hospital admission records, missing all cases that never reach a hospital.

e) A sociologist studies political attitudes by surveying people who answer their phones, missing the attitudes of people who screen calls.

f) A climate scientist models future climate using atmospheric data from weather stations concentrated in developed nations, with sparse coverage in developing nations.

g) An investor evaluates companies using quarterly earnings reports, ignoring employee morale, supply chain resilience, and institutional knowledge.

h) A literary critic studies the canon of "great literature" -- which is overwhelmingly white, male, and Western -- and draws conclusions about the nature of literary greatness.

A2. For each of the following claims, identify what data was likely measured to produce the claim and what data was likely missing. Assess how the missing data might change the conclusion.

a) "Crime in this neighborhood has dropped 30% since we increased police patrols."

b) "Our customers are very satisfied -- our survey shows a 4.5/5 average rating."

c) "Ancient Rome was the most advanced civilization of its era."

d) "This drug is safe -- clinical trials showed no significant side effects."

e) "Social media use is correlated with depression in teenagers."

f) "GDP growth has been strong, so the economy is healthy."

g) "Our algorithm has 95% accuracy in predicting recidivism."

h) "Remote work reduces productivity -- output metrics declined during the pandemic."

A3. Classify each of the following as primarily a measurement availability problem, an institutional incentive problem, or a path dependence problem. Some may involve more than one; identify the primary driver.

a) Psychology's WEIRD problem

b) The 10/90 gap in global health research

c) Valley bias in archaeology

d) Body count metrics in Vietnam

e) GDP as the primary measure of national success

f) Facial recognition algorithms trained on non-representative datasets

g) Hot-spot policing feedback loops

h) The dominance of English-language sources in academic research

A4. For each domain in the chapter, identify one question that the streetlight effect has prevented the field from answering. Explain why the question remains unanswered and what it would take to answer it.

a) Research methodology

b) Policing

c) Archaeology

d) Data science

e) Medicine

f) Economics

A5. The McNamara Fallacy has four steps: (1) measure what is easy, (2) disregard what is hard to measure, (3) presume the hard-to-measure is not important, (4) presume the hard-to-measure does not exist. Identify a contemporary example in each of the following fields and trace all four steps:

a) Education

b) Healthcare administration

c) Corporate management

d) Environmental policy

e) Social media platform design

Part B: Analysis

These exercises require deeper analysis of streetlight effect dynamics.

B1. Streetlight Mapping. Choose a field you know well (your academic discipline, your industry, your area of policy interest) and map its streetlight effect comprehensively.

a) What are the field's primary data sources? What makes these data sources convenient?

b) What questions does the field study most frequently? Are these the most important questions, or the most measurable ones?

c) What populations, phenomena, or variables are systematically understudied? What makes them hard to study?

d) What institutional incentives (funding structures, publication norms, career advancement criteria) reinforce the streetlight effect?

e) What path dependencies (established methods, training pipelines, journal standards) make it difficult to shift the field's attention to what is in the dark?

f) What would it cost -- in money, time, career risk, and institutional change -- to redirect the field's attention toward the dark areas?

B2. Dark Data Audit. Choose a specific dataset, report, or body of research that you use or encounter regularly.

a) Identify the data that is present. What was collected? From whom? By what method? Under what conditions?

b) Identify the dark data -- the data that is absent. What was not collected? Who was not sampled? What variables were not measured? What time periods or geographies are not represented?

c) Assess the gap. Could the missing data change the conclusions drawn from the present data? In what direction?

d) Design a minimal intervention -- the smallest, cheapest, most practical change -- that would reduce the most consequential gap between present data and missing data.

B3. Feedback Loop Analysis. The chapter describes how hot-spot policing creates a feedback loop: policing generates data, data directs policing, more policing generates more data.

a) Identify three other domains where the streetlight effect operates through a feedback loop. For each, trace the loop.

b) For each loop you identified, assess whether the loop is self-limiting (does it eventually correct itself?) or self-reinforcing (does it grow stronger over time?).

c) What would break each loop? Identify a structural intervention, not just an individual awareness.

B4. Countermeasure Evaluation. The chapter describes five countermeasures: dark data awareness, triangulation, mixed methods, deliberately studying the hard-to-measure, and searching for absence.

a) For each countermeasure, identify a specific case where it has been successfully applied. What made it work?

b) For each countermeasure, identify a limitation. What can it not solve? Under what conditions does it fail?

c) Rank the five countermeasures by feasibility (how easy they are to implement) and by effectiveness (how much they reduce the streetlight bias). Are the most feasible also the most effective? If not, why not?

B5. McNamara Fallacy Diagnosis. Choose an institution you are familiar with (your workplace, your university, a government agency, a nonprofit). Identify a metric that the institution uses as a primary measure of success.

a) What does the metric actually measure? What does it claim to measure?

b) What important aspects of the institution's mission are not captured by the metric?

c) Has the metric become a target? Has behavior changed to optimize the metric rather than the underlying goal? (Connect to Goodhart's Law from Chapter 15.)

d) Design a supplementary metric or evaluation process that would capture what the primary metric misses. What would be the obstacles to implementing it?

Part C: Application

These exercises ask you to apply streetlight effect concepts to your own experience.

C1. Personal Streetlight Inventory. Identify three decisions you have made in the past year that were influenced by the streetlight effect -- where you searched for information or evaluated options based on what was easily available rather than what was most relevant.

a) For each, what was the "streetlight" (the easy source of information)?

b) What was the "park" (the harder-to-access but more relevant source)?

c) Did the streetlight bias affect the quality of your decision? How?

d) What would you do differently if you recognized the streetlight effect in real time?

C2. Professional Streetlight Audit. In your professional field or area of study:

a) What is the equivalent of the WEIRD problem? What population or context is overrepresented in the field's knowledge base?

b) What is the equivalent of GDP? What metric is treated as a proxy for success even though it misses important dimensions?

c) What is the equivalent of the dark figure of crime? What important phenomena are systematically underreported or unmeasured?

d) What would a "dark data audit" of your field reveal? What would you find if you looked for what is not being studied?

C3. Measurement Design Challenge. You are asked to design a measurement system for something important but hard to measure. Choose one:

a) Community wellbeing in a city

b) The quality of education at a university

c) The health of a workplace culture

d) The ecological health of a watershed

e) The quality of a romantic relationship

For your chosen topic: (i) Identify what is easy to measure and what is hard to measure. (ii) Design a measurement system that includes both. (iii) Anticipate how the measurement system might create its own streetlight effect -- what aspects of the topic might still be left in the dark?

C4. Searching for Absence. Choose a topic you know well and practice the "searching for absence" countermeasure.

a) What is the standard body of evidence on this topic? What sources are typically cited?

b) Who is not represented in this evidence? What perspectives, populations, or experiences are missing?

c) What questions are not being asked? What topics are considered outside the scope of the standard evidence?

d) If you could add one missing piece of evidence to the standard body, what would it be? Why?

Part D: Synthesis

These exercises require integrating streetlight effect concepts with ideas from earlier chapters.

D1. Streetlight Effect and Signal/Noise (Ch. 6). Chapter 6 examined the challenge of separating signal from noise in data.

a) How does the streetlight effect interact with signal-to-noise problems? If the data is biased (streetlight effect) and noisy (signal/noise), how do the two distortions compound?

b) Can sophisticated signal-processing techniques compensate for the streetlight effect? Under what conditions might they make it worse?

c) The chapter argues that the streetlight effect is "not a noise problem but a missing-signal problem." Explain this distinction and its implications for data analysis.

D2. Streetlight Effect and the McNamara Fallacy (Ch. 15). Chapter 15 examined how metrics become targets and targets corrupt behavior.

a) Is the streetlight effect a cause of the McNamara Fallacy, a consequence of it, or both? Trace the causal relationship.

b) Can a measurement system be designed that avoids both the streetlight effect and Goodhart's Law? What would it look like? What tradeoffs would be involved?

c) The chapter describes the McNamara Fallacy as having four steps. At which step is intervention most effective? How would you design an institutional process that interrupts the fallacy before step 3?

D3. Streetlight Effect and Feedback Loops (Ch. 2). The chapter identifies feedback loops in hot-spot policing and in the self-reinforcing nature of the streetlight effect generally.

a) Is the streetlight effect fundamentally a positive feedback loop? Draw the causal loop diagram.

b) What negative feedback mechanisms, if any, exist to counteract the streetlight effect? How effective are they?

c) Chapter 2 discussed how positive feedback loops can be interrupted by introducing balancing feedback. Design a balancing feedback mechanism that would counteract the streetlight effect in a domain of your choice.

D4. Streetlight Effect and Senescence (Ch. 31). The chapter draws a connection between the streetlight effect and institutional senescence.

a) How does the streetlight effect contribute to the aging of knowledge systems? When a field has been searching under the streetlight for decades, in what sense has its knowledge "senesced"?

b) Chapter 31's threshold concept was "Aging as Accumulated Compromise." How does this apply to the streetlight effect? Is the streetlight effect itself a form of accumulated compromise?

c) Can a field "rejuvenate" its knowledge by deliberately redirecting attention to the dark? What would this rejuvenation look like in practice? Identify a real case where this has happened.

D5. Streetlight Effect and the Lifecycle S-Curve (Ch. 33). The chapter suggests that the streetlight effect has its own S-curve dynamics.

a) Map the S-curve of the streetlight effect in a specific field. When did the field first begin searching under the streetlight? When did the bias become entrenched? Is it currently in the saturation or decline phase?

b) What would succession look like for the streetlight effect -- a new methodological paradigm replacing the old one? Has this happened in any of the domains discussed in the chapter?

c) Can the concept of stacked S-curves (Ch. 33) be applied to countermeasures? Could a field sustain progress against the streetlight effect by launching successive waves of methodological innovation?

Part E: Advanced

These exercises push into territory beyond the chapter's explicit coverage.

E1. The Streetlight Effect in Everyday Life. The chapter focuses on institutional and professional domains. But the streetlight effect operates in personal life too. Identify three ways the streetlight effect shapes personal decision-making -- in career choices, relationship evaluations, health decisions, or other everyday domains. For each, describe the streetlight, the park, and the structural force that keeps the search under the light. Design a personal countermeasure for each.

E2. The Meta-Streetlight. The chapter itself is subject to the streetlight effect. It examines domains where the streetlight effect has been documented and discussed -- precisely because documentation and discussion make them visible. What domains might exhibit the streetlight effect but have not been studied enough to make it into a chapter like this one? Identify at least three candidates and explain why the streetlight effect in those domains has itself remained in the dark.

E3. The Ethics of the Streetlight. The streetlight effect has distributional consequences: some populations are studied and served; others are not. Develop an ethical framework for evaluating streetlight effects. When is the streetlight effect merely suboptimal (it would be better to search the dark, but the cost is high)? When is it unjust (the dark areas are systematically those that affect marginalized or powerless populations)? How should responsibility be allocated -- to individual researchers, to institutions, to funders, to policymakers?

E4. Constructive Streetlights. The chapter presents the streetlight effect as a distortion. But is it always bad? Are there cases where searching under the streetlight is the right strategy -- where the value of finding something under the light exceeds the value of searching in the dark? Develop criteria for when the streetlight effect is harmful and when it is acceptable. What factors determine the threshold?

E5. Designing Dark Institutions. If the streetlight effect is structural -- driven by measurement availability, institutional incentives, and path dependence -- then counteracting it requires structural change, not just individual awareness. Design an institutional structure (a funding agency, a research institute, a policy evaluation body) that is explicitly designed to resist the streetlight effect. What would its incentives, methods, hiring criteria, and evaluation standards look like? What trade-offs would it face?

Part M: Mixed Practice (Interleaved)

These exercises deliberately mix concepts from the current chapter with concepts from Chapters 31 and 33 for spaced review.

M1. Chapter 31 described how systems age through the accumulation of individually rational compromises. The streetlight effect accumulates through individually rational choices (each researcher studies what is convenient). Compare the aging trajectory of an institution's streetlight bias with the senescence patterns described in Chapter 31. At what point does the accumulated bias become unserviceable? What triggers the crisis?

M2. Chapter 33 described the carrying capacity as the limit that causes an S-curve to flatten. What is the "carrying capacity" of research under the streetlight? Is there a limit to how much can be learned by studying only what is convenient? When is that limit reached, and what happens next?

M3. Chapter 31 distinguished between programmed senescence (built-in timers) and damage-accumulation senescence (entropy-driven degradation). Is the streetlight effect more like programmed senescence (built into the structure of modern research institutions) or damage accumulation (a gradual drift caused by accumulated individual choices)? Or both?

M4. Chapter 33 argued that the only way to sustain growth is to stack S-curves -- launching new growth initiatives before the current one peaks. Can a field sustain knowledge growth by stacking methodological S-curves -- adopting new methods (LIDAR in archaeology, cross-cultural sampling in psychology, alternative metrics in economics) before the current methods' returns diminish? Identify a field where this stacking has occurred and assess its success.

M5. Combine the streetlight effect (Ch. 35) with senescence (Ch. 31) and the S-curve (Ch. 33) into a unified diagnostic for a knowledge field you care about. Write a one-page "methodological health report" that assesses: (i) Where is the field searching (under what streetlight)? (ii) What is in the dark? (iii) What accumulated biases have senesced into the field's methods? (iv) Where is the field on the S-curve of its current methodological paradigm? (v) What would the next S-curve look like?