Quiz: Precision Without Accuracy

Q: In the archery analogy, "Archer C" (precise but not accurate) represents: A measurement that varies widely but averages to the correct answer A measurement that consistently produces the same wrong answer A measurement that is both consistent and correct A measurement that varies widely and averages to a wrong answer

B) Tight clustering far from the target — consistently, confidently wrong. Reference: Section 12.1

Q: The "specificity heuristic" means: People prefer specific, precise claims over vague ones — regardless of evidence People are naturally good at estimating specific quantities Specific claims are always more accurate than vague ones Precision requires specificity

A) Precision is interpreted as credibility, independent of accuracy. Reference: Section 12.3

Q: The VaR models at Lehman Brothers failed because: The mathematics was incorrect The models assumed normal distributions, which dramatically underestimate extreme events The analysts were incompetent The models were never calibrated

B) The Gaussian assumption produced precise but systematically wrong risk estimates. Reference: Section 12.5

Q: "Error propagation" means: Errors spread from one department to another When imprecise measurements are combined, the imprecision compounds Errors are always amplified by institutional processes Measurement error increases over time

B) Combining imprecise inputs through calculation produces results less accurate than any individual input. Reference: Section 12.6

Q: The IQ score example demonstrates precision without accuracy because: IQ tests are invalid The single-number score implies unit-level precision while the actual measurement has a standard error of 3-5 points IQ changes over time Different IQ tests give different results

B) The reported precision (to the integer) far exceeds the measurement's actual precision (±3-5 points). Reference: Section 12.2

Q: "If a number is reported with many decimal places, the underlying measurement must be that precise."

False. Decimal places in a reported number reflect the arithmetic precision of the calculation, not the accuracy of the measurement. A GDP forecast reported as 2.37% has arithmetic precision to 0.01% but measurement accuracy of ±1-2%.

Q: "The solution to precision without accuracy is to stop using numbers entirely."

False. The solution is to report numbers with their uncertainty — confidence intervals, ranges, uncertainty budgets — so that the precision of the number is calibrated to the accuracy of the measurement.

Q: "Qualitative assessments are always less objective than quantitative ones."

False. Objectivity and precision are not the same thing. A qualitative assessment based on systematic observation can be more objective (and more accurate) than a precise number produced by a model with unverifiable assumptions.

Quiz: Precision Without Accuracy

Target: 70% or higher to proceed confidently.

Section 1: Multiple Choice (1 point each)

1. In the archery analogy, "Archer C" (precise but not accurate) represents: - A) A measurement that varies widely but averages to the correct answer - B) A measurement that consistently produces the same wrong answer - C) A measurement that is both consistent and correct - D) A measurement that varies widely and averages to a wrong answer

Answer

**B)** Tight clustering far from the target — consistently, confidently wrong. *Reference:* Section 12.1

2. The "specificity heuristic" means: - A) People prefer specific, precise claims over vague ones — regardless of evidence - B) People are naturally good at estimating specific quantities - C) Specific claims are always more accurate than vague ones - D) Precision requires specificity

Answer

**A)** Precision is interpreted as credibility, independent of accuracy. *Reference:* Section 12.3

3. The VaR models at Lehman Brothers failed because: - A) The mathematics was incorrect - B) The models assumed normal distributions, which dramatically underestimate extreme events - C) The analysts were incompetent - D) The models were never calibrated

Answer

**B)** The Gaussian assumption produced precise but systematically wrong risk estimates. *Reference:* Section 12.5

4. "Error propagation" means: - A) Errors spread from one department to another - B) When imprecise measurements are combined, the imprecision compounds - C) Errors are always amplified by institutional processes - D) Measurement error increases over time

Answer

**B)** Combining imprecise inputs through calculation produces results less accurate than any individual input. *Reference:* Section 12.6

5. The IQ score example demonstrates precision without accuracy because: - A) IQ tests are invalid - B) The single-number score implies unit-level precision while the actual measurement has a standard error of 3-5 points - C) IQ changes over time - D) Different IQ tests give different results

Answer

**B)** The reported precision (to the integer) far exceeds the measurement's actual precision (±3-5 points). *Reference:* Section 12.2

Section 2: True/False with Justification (1 point each)

6. "If a number is reported with many decimal places, the underlying measurement must be that precise."

Answer

**False.** Decimal places in a reported number reflect the arithmetic precision of the *calculation*, not the accuracy of the *measurement*. A GDP forecast reported as 2.37% has arithmetic precision to 0.01% but measurement accuracy of ±1-2%.

7. "The solution to precision without accuracy is to stop using numbers entirely."

Answer

**False.** The solution is to report numbers *with their uncertainty* — confidence intervals, ranges, uncertainty budgets — so that the precision of the number is calibrated to the accuracy of the measurement.

8. "Qualitative assessments are always less objective than quantitative ones."

Answer

**False.** Objectivity and precision are not the same thing. A qualitative assessment based on systematic observation can be more objective (and more accurate) than a precise number produced by a model with unverifiable assumptions.

Section 3: Short Answer (2 points each)

9. Explain why the institutional demand for legible numbers systematically produces false precision.

Sample Answer

Institutions need processable inputs for decision-making: numbers that can be compared, aggregated, tracked, and reported. Qualitative assessments ("the risk is uncertain") cannot be processed by institutional machinery requiring numerical inputs. This demand forces quantification → quantification forces precision → precision forces a false sense of certainty. The result: institutions systematically prefer precise wrong numbers over honest uncertain assessments, because the wrong numbers are institutionally legible and the honest assessments are not.

Section 4: Applied Scenario (3 points)

10. A company uses a customer satisfaction score (1-10 scale, reported to one decimal place) to evaluate its 50 retail stores. Store A scores 7.3; Store B scores 7.1. The company plans to implement a performance improvement program at Store B (the lower-scoring store). Apply the precision-accuracy framework to evaluate this decision.

Sample Answer

The decision assumes the 0.2-point difference is meaningful. But: (1) the measurement error of customer satisfaction surveys is typically ±0.3-0.5 points, so 7.3 and 7.1 are statistically indistinguishable. (2) The score aggregates diverse customer experiences into a single dimension, masking important variation. (3) The score at any store varies week-to-week based on factors unrelated to store quality (weather, seasonal patterns, sample composition). (4) Goodhart's Law predicts that targeting the score will produce score optimization rather than genuine improvement. Recommendation: Don't make consequential decisions based on differences within the measurement's error margin. Report scores as ranges (Store A: ~7.0-7.5; Store B: ~6.8-7.4). Use qualitative assessment (mystery shoppers, observational audits) alongside the quantitative score.

Scoring & Next Steps

Score	Assessment	Recommended Action
< 50%	Needs review	Re-read 12.1–12.3
50–70%	Partial	Review the VaR case and the archery analogy
70–85%	Solid	Ready to proceed
> 85%	Strong	Proceed to Chapter 13