Chapter 35 Quiz: Self-Assessment

Instructions: Answer each question without looking back at the chapter. After completing all questions, check your answers against the key at the bottom. If you score below 70%, revisit the relevant sections before moving on to Chapter 36.

Multiple Choice

Q1. The streetlight effect is best defined as:

a) The tendency to search for information only at night when visibility is poor b) The systematic bias toward searching where observation is easy, data is available, or methods are well-developed -- regardless of whether those areas are likely to contain what is sought c) The tendency to look for confirming evidence rather than disconfirming evidence d) The bias introduced by using artificial lighting in laboratory settings

Q2. The McNamara Fallacy, as described in this chapter, proceeds through four steps. The correct order is:

a) Measure the important → disregard the unimportant → make decisions → evaluate results b) Measure what is easy → disregard what is hard to measure → presume the hard-to-measure is not important → presume the hard-to-measure does not exist c) Identify the problem → collect data → analyze data → implement solutions d) Define metrics → optimize metrics → report metrics → celebrate metrics

Q3. The WEIRD problem in psychology refers to:

a) The tendency for psychological experiments to produce unexpected results b) The overrepresentation of Western, Educated, Industrialized, Rich, and Democratic subjects in published research, despite this population being unrepresentative of global humanity c) The difficulty of replicating psychological studies across different laboratories d) The ethical problems created by using deception in psychological research

Q4. The "dark figure of crime" refers to:

a) Criminal activity that occurs at night b) The gap between crimes committed and crimes recorded -- the offenses that are never reported, investigated, or entered into official statistics c) Crimes committed by unknown perpetrators d) The margin of error in crime statistics

Q5. Hot-spot policing creates a feedback loop because:

a) High crime areas always produce more crime over time b) Police presence deters crime, which reduces data, which reduces police presence c) Policing generates data, data directs more policing to the same areas, which generates more data from those areas, while under-policed areas generate less data and receive less attention d) Community trust increases in heavily policed areas, leading to more crime reporting

Q6. Valley bias in archaeology refers to:

a) The tendency to find more artifacts in river valleys because ancient people preferred valleys b) The systematic overrepresentation of river valley and coastal plain sites in the archaeological record, driven by the greater accessibility and lower cost of excavating in these locations c) The distortion caused by valley flooding destroying archaeological evidence d) The academic disagreement between archaeologists working in different geographical regions

Q7. The data availability bias in big data and machine learning is particularly dangerous because:

a) Big datasets always contain errors b) The sheer volume of data creates an illusion of completeness, suppressing uncertainty signals that would otherwise indicate the conclusions might be wrong c) Machine learning algorithms are inherently unreliable d) Big data is too expensive for most researchers to access

Q8. Neglected tropical diseases illustrate the streetlight effect in medicine because:

a) They are difficult to diagnose b) They primarily affect populations in low-income countries, where the funding, infrastructure, and pharmaceutical market incentives that drive medical research are largely absent c) They require specialized tropical laboratories to study d) They are caused by organisms that are hard to observe under microscopes

Q9. GDP fails as a measure of national wellbeing primarily because:

a) It is calculated using inaccurate data b) It counts market-exchanged goods and services while ignoring unpaid labor, environmental degradation, and non-market dimensions of wellbeing -- making it a streetlight that illuminates economic activity while leaving human welfare in the dark c) Different countries calculate it using different methods d) It does not account for inflation

Q10. The three structural forces that create the streetlight effect, as identified in this chapter, are:

a) Laziness, incompetence, and corruption b) Funding limitations, time constraints, and publication pressure c) Measurement availability, institutional incentives, and path dependence d) Data scarcity, methodological disagreement, and political interference

Q11. The Data Availability Principle states that:

a) More data always leads to better conclusions b) Data should always be made publicly available c) The conclusions of any inquiry are bounded by the representativeness of the data on which they are based, and systematically non-representative data produces systematically distorted conclusions d) Data availability determines the cost of research

Q12. David Hand's concept of "dark data" refers to:

a) Data that has been encrypted or hidden b) Data about criminal activity c) Data that does not exist in any dataset -- not because the phenomena it would describe do not exist, but because no one has collected, measured, or recorded it d) Data from the dark web

Q13. Triangulation as a countermeasure to the streetlight effect involves:

a) Using three data points to establish a trend b) Approaching a question from multiple independent methodological directions and checking whether conclusions converge c) Dividing the research area into triangular sections d) Prioritizing the three most important variables in any study

Q14. The chapter's threshold concept -- Measurement Creates Its Own Reality -- means:

a) Quantum measurement affects the phenomena being measured b) The act of measuring something changes its value c) The choice of what to measure does not merely describe reality but actively shapes what counts as knowledge, what receives attention, what gets funded, and ultimately what gets done d) All measurements are inherently subjective

Q15. The connection between the streetlight effect and Goodhart's Law (from Ch. 15) is:

a) They are unrelated concepts from different fields b) Goodhart's Law causes the streetlight effect c) The streetlight effect is the deepest version of the pattern: letting what is measurable define what is important. The McNamara Fallacy is the streetlight effect applied to management. Goodhart's Law is the McNamara Fallacy applied to incentive design. d) They are different names for the same concept

Q16. The chapter argues that the streetlight effect differs from simple laziness because:

a) It affects intelligent people more than unintelligent people b) It is structural -- driven by measurement availability, institutional incentives, and path dependence that make the bias self-reinforcing even when individuals are aware of it c) It only affects institutions, not individuals d) It is always intentional

Q17. LIDAR technology's impact on archaeology illustrates:

a) The limits of traditional archaeological methods in revealing what was hidden by the streetlight effect -- showing vast urban complexes in forests that were invisible to ground-based methods b) The superiority of technology over traditional methods in all cases c) The dangers of relying on remote sensing instead of excavation d) The cost-effectiveness of modern archaeological methods

Q18. Mixed methods research counteracts the streetlight effect by:

a) Using multiple quantitative methods simultaneously b) Combining quantitative approaches (strong on measurement, weak on context) with qualitative approaches (strong on context, weak on measurement) to illuminate both what the streetlight reveals and what it misses c) Requiring all research to include both laboratory and field components d) Ensuring that both Eastern and Western research methods are used

Q19. The chapter identifies the streetlight effect as self-reinforcing because:

a) People who search under the streetlight become more confident in their results b) Each round of research under the streetlight builds infrastructure (methods, datasets, expertise) that makes the next round of research in the same area easier, while doing nothing to make research in the dark areas easier c) Researchers who find results under the streetlight are more likely to continue searching there d) Both b and c

Q20. The chapter argues that the streetlight effect and survivorship bias (Ch. 37) form a "double filter" on human knowledge because:

a) They are caused by the same cognitive bias b) First we search in biased locations (streetlight effect), then from those biased locations we notice only the salient results (survivorship bias) -- compounding the distortion c) They cancel each other out d) They are both forms of confirmation bias

Answer Key

  1. b -- Section 35.1. The streetlight effect is the systematic bias toward searching where observation is easy, not where the answer is likely.

  2. b -- Section 35.2. The McNamara Fallacy moves from measuring the easy to presuming the unmeasured does not exist.

  3. b -- Section 35.2. WEIRD = Western, Educated, Industrialized, Rich, Democratic. This population is an outlier, not a representative sample.

  4. b -- Section 35.3. The dark figure is the gap between crimes committed and crimes recorded.

  5. c -- Section 35.3. The feedback loop: policing → data → more policing → more data, while under-policed areas stay dark.

  6. b -- Section 35.4. Valley bias is driven by accessibility and cost, not just by where ancient people lived.

  7. b -- Section 35.5. Volume creates the illusion that the data is comprehensive, masking systematic gaps.

  8. b -- Section 35.6. The light shines where the money is; NTDs are in the funding dark.

  9. b -- Section 35.7. GDP measures market activity but ignores unpaid labor, environmental costs, and wellbeing.

  10. c -- Section 35.8. Measurement availability, institutional incentives, and path dependence.

  11. c -- Section 35.8. Conclusions are bounded by data representativeness.

  12. c -- Section 35.9. Dark data is uncollected data about real phenomena.

  13. b -- Section 35.9. Triangulation means approaching from multiple independent directions.

  14. c -- Section 35.10. Measurement does not describe reality neutrally; it shapes what counts as real.

  15. c -- Section 35.2. They form a nested hierarchy of the same structural error at different scales.

  16. b -- Section 35.8. The bias is structural and self-reinforcing, not a personal failing.

  17. a -- Section 35.4. LIDAR revealed what was hidden by traditional archaeology's streetlight.

  18. b -- Section 35.9. Mixed methods combine the strengths of quantitative and qualitative approaches.

  19. d -- Section 35.8. Both the infrastructure effect and the behavioral tendency are self-reinforcing.

  20. b -- Section 35.8. Streetlight effect biases the search; survivorship bias biases what we see in the results. Together, they compound.

Scoring Guide

  • 18-20 correct (90-100%): Excellent. You have a thorough grasp of the streetlight effect and its cross-domain manifestations. Proceed to Chapter 36.
  • 14-17 correct (70-85%): Good. You understand the core concepts. Review any sections corresponding to questions you missed before proceeding.
  • 10-13 correct (50-65%): Fair. Revisit Sections 35.2, 35.5, and 35.8-35.10, which contain the chapter's most important structural arguments.
  • Below 10 (under 50%): Re-read the chapter before proceeding. Focus especially on the three structural forces (Section 35.8), the countermeasures (Section 35.9), and the threshold concept (Section 35.10).