Quiz: Chapter 34

The Business of Data Science

Instructions: Answer all questions. Multiple-choice questions have one correct answer unless otherwise stated. Short-answer questions should be answered in 2-4 sentences.

Question 1 (Multiple Choice)

A churn model has the following per-prediction economics: TP value = $78, FP cost = $30, FN cost = $180, TN value = $0. What is the break-even precision?

  • A) 0.30
  • B) 0.28
  • C) 0.70
  • D) 0.43

Answer: B) 0.28. Break-even precision = FP cost / (TP value + FP cost) = $30 / ($78 + $30) = $30 / $108 = 0.278, which rounds to 0.28. At this precision, the value gained from true positives exactly offsets the cost of false positives. Any precision above 0.28 generates net positive value. This means the model has a large margin of safety --- it would need to become dramatically worse before it should be turned off.

Question 2 (Multiple Choice)

A data scientist presents model results to the chief medical officer with a 40-slide deck covering data cleaning, feature engineering, model selection, hyperparameter tuning, and evaluation metrics. The CMO loses interest by slide 8. What is the primary mistake?

  • A) The deck should have been shorter (10 slides maximum)
  • B) The presentation followed the data scientist's workflow instead of leading with the business recommendation
  • C) The CMO is not technical enough to understand ML
  • D) The data scientist should have used more visualizations

Answer: B) The presentation followed the data scientist's workflow instead of leading with the business recommendation. The Pyramid Principle says: lead with the answer, support with key evidence, provide technical detail only if asked. Stakeholders care about "what should we do and why," not "how I built the model." The deck length is a symptom, not the root cause --- even a 10-slide deck that starts with data cleaning is structured incorrectly.

Question 3 (Short Answer)

Explain why optimizing a classification threshold for accuracy (maximizing the number of correct predictions) often produces a different threshold than optimizing for business value. Use a specific example.

Answer: Accuracy treats all errors equally --- a false positive and a false negative both count as one mistake. Business value weights errors by their dollar impact. In churn prediction, a false negative (missed churner) costs $180 while a false positive (unnecessary outreach) costs $30. Maximizing accuracy pushes the threshold toward 0.50, which minimizes total errors. Maximizing business value pushes the threshold lower (toward 0.15--0.20), accepting more false positives to catch more true positives, because each caught churner is worth six times more than each false alarm costs.

Question 4 (Multiple Choice)

An executive says: "Our competitors are using AI. We need AI too." Which of the following is the most productive first response from the data science team?

  • A) "Let me research what AI tools are available and present options next week."
  • B) "What decision are you trying to improve?"
  • C) "We should start by hiring three more data scientists."
  • D) "AI is overhyped. We should focus on basic analytics first."

Answer: B) "What decision are you trying to improve?" This grounds the conversation in business value rather than technology hype. AI is a tool for improving decisions. If the executive cannot name a specific decision, the conversation is about branding, not technology, and no amount of hiring or tool selection will produce value. The other options either jump to solutions (A, C) or are dismissive (D).

Question 5 (Multiple Choice)

A company's data maturity assessment reveals: data is in a centralized warehouse, BI dashboards exist, but there is no ML model in production, no A/B testing culture, and no data quality automation. What maturity level is this, and what should the first ML initiative focus on?

  • A) Level 1 (Reactive); invest in data infrastructure
  • B) Level 2 (Descriptive); start with a high-ROI predictive model
  • C) Level 3 (Predictive); invest in MLOps
  • D) Level 2 (Descriptive); build a recommendation engine

Answer: B) Level 2 (Descriptive); start with a high-ROI predictive model. The company has the data infrastructure (centralized warehouse, BI dashboards) but has not yet deployed ML. The first initiative should target a use case with clear ROI and a straightforward feedback loop --- like churn prediction or lead scoring --- not a complex system like a recommendation engine, which requires more mature data pipelines and experimentation infrastructure.

Question 6 (Short Answer)

An A/B test shows a +0.4% lift in conversion rate with a p-value of 0.23. The product VP wants to launch the new feature anyway. Explain what "p = 0.23" means in plain language, and describe one constructive alternative to either launching or killing the feature.

Answer: A p-value of 0.23 means that if the new feature had zero true effect, there would be a 23% chance of seeing a result this large or larger by random chance alone. In plain language: the observed improvement could easily be noise. One constructive alternative is to extend the test with a larger sample size to increase statistical power. If the 0.4% effect is real, a larger test will confirm it with confidence. Another option is a staged rollout to 10% of users with close monitoring.

Question 7 (Multiple Choice)

Which of the following is the best title for a stakeholder presentation slide showing the churn model's performance?

  • A) "Model Performance Metrics"
  • B) "Confusion Matrix and ROC Curve"
  • C) "The model catches 72% of churners with 70% flag accuracy"
  • D) "XGBoost Results: AUC = 0.88, F1 = 0.71"

Answer: C) "The model catches 72% of churners with 70% flag accuracy." Slide titles should state the takeaway, not the topic. Option C tells the audience the conclusion in business language. Option A is a topic label (says nothing about performance). Option B names technical artifacts. Option D uses technical metrics that non-technical stakeholders will not understand. The audience should grasp the message without reading the slide body.

Question 8 (Short Answer)

A data scientist calculates that their churn model saves $47,000 per month. The CFO says: "That number assumes a 60% intervention success rate. If it is 40%, what happens?" Explain why sensitivity analysis is important and what the data scientist should have prepared in advance.

Answer: Sensitivity analysis is important because ROI calculations depend on assumptions, and assumptions are often wrong. The data scientist should have prepared a tornado chart showing how ROI changes as each key assumption varies by a reasonable range (e.g., plus or minus 25%). For the intervention success rate specifically, the data scientist should know the break-even success rate --- the minimum rate at which the model remains profitable. Presenting the ROI as a range ("$28K--$65K per month depending on intervention success") is more credible than presenting a single point estimate.

Question 9 (Multiple Choice)

Which of the following is NOT a component of a minimal model governance framework?

  • A) Model registry tracking all production models and their versions
  • B) Fairness audit before deployment
  • C) Requirement that all models achieve AUC > 0.90 before deployment
  • D) Retirement policy defining when a model should be decommissioned

Answer: C) Requirement that all models achieve AUC > 0.90 before deployment. A blanket AUC threshold is not governance --- it is an arbitrary performance bar that ignores domain context. A fraud model with AUC 0.85 might be enormously valuable; a churn model with AUC 0.92 might be worthless if nobody acts on the predictions. Governance is about processes (registry, approval, monitoring, fairness, documentation, retirement), not about mandating a single metric threshold.

Question 10 (Multiple Choice)

A junior data scientist's portfolio project says: "Trained XGBoost on Kaggle churn data. Achieved 81% accuracy." What is the single most important addition that would demonstrate business thinking?

  • A) Adding a SHAP analysis of feature importance
  • B) Including an ROI calculation showing the dollar value of the model's predictions
  • C) Using a more complex model (neural network) to achieve higher accuracy
  • D) Adding more visualizations to the EDA section

Answer: B) Including an ROI calculation showing the dollar value of the model's predictions. The ROI calculation demonstrates that the data scientist understands why the model matters, not just how it works. It requires assigning dollar values to each cell of the confusion matrix, computing net value, and comparing to a baseline. This is the skill that separates "I can build models" from "I can deliver business value." SHAP (A) is useful but still technical. A more complex model (C) adds nothing. More EDA (D) is cosmetic.

Question 11 (Short Answer)

Explain the difference between a data-driven decision and a data-informed decision. When is each appropriate?

Answer: A data-driven decision is one where the data determines the action --- if the A/B test says version B wins, you ship version B. A data-informed decision is one where data is an input alongside other considerations (strategy, customer feedback, competitive pressure, ethical concerns). Most business decisions should be data-informed, not data-driven. Data-driven decisions are appropriate for high-volume, repeatable, low-stakes decisions (ad bidding, email send-time optimization). Data-informed decisions are appropriate for strategic, high-stakes, or value-laden decisions where context beyond the data matters.

Question 12 (Multiple Choice)

You are presenting the hospital readmission model to the chief medical officer. The model has recall = 0.68, meaning it catches 68% of readmissions. Which framing is most effective?

  • A) "The model has a recall of 0.68 and a false negative rate of 0.32."
  • B) "The model catches about two-thirds of readmissions --- better than the current 40% catch rate from clinical intuition, but not a replacement for clinical judgment."
  • C) "The model misses 32% of readmissions, which is a significant limitation."
  • D) "The model's AUC is 0.81, which indicates good discriminative performance."

Answer: B) "The model catches about two-thirds of readmissions --- better than the current 40% catch rate from clinical intuition, but not a replacement for clinical judgment." This framing provides context (comparison to baseline), uses accessible language (no technical terms), sets realistic expectations (not a replacement), and is honest about the limitation without leading with the negative. Option A uses technical jargon. Option C leads with the weakness, undermining confidence. Option D uses a metric the CMO does not understand.

This quiz covers Chapter 34: The Business of Data Science. Return to the chapter for full context.