Exercises: Chapter 34

The Business of Data Science

Exercise 1: Expected Value Framework (Conceptual + Code)

A fraud detection model at an online payments company produces the following confusion matrix on one month of production data (200,000 transactions):

The economics: - Each caught fraud (TP) saves the company $850 in chargebacks and investigation costs. The review cost per flagged transaction is $15. - Each false alarm (FP) incurs a $15 review cost plus a $5 customer friction cost (delayed transaction, support call). - Each missed fraud (FN) costs $850 in chargebacks, plus $200 in customer trust damage. - True negatives have zero cost.

a) Compute the net value per TP, FP, FN, and TN.

b) Compute the total monthly value of the model.

c) What is the model's break-even precision? Is the current precision above or below break-even?

d) The compliance team wants to increase the threshold to reduce false positives by 50%. Assuming this reduces TPs by 20% and FPs by 50%, compute the new monthly value. Is this a good business decision?

import numpy as np
from sklearn.metrics import confusion_matrix

# Use the compute_model_roi function from the chapter.
# Adapt the parameters for the fraud detection scenario.
np.random.seed(42)

# Your code here

Exercise 2: Threshold Optimization for Three Domains (Code)

Using the optimize_threshold_for_value function from the chapter, find the optimal threshold for the following three scenarios. For each, simulate 10,000 observations with a plausible class balance and model performance (AUC approximately 0.80--0.85).

Scenario A --- SaaS Churn: TP value = $78, FP cost = $30, FN cost = $180, base rate = 12%.

Scenario B --- Manufacturing Defect Detection: TP value = $5,000 (defect caught before shipment), FP cost = $200 (unnecessary inspection), FN cost = $45,000 (defective product shipped, recall costs), base rate = 2%.

Scenario C --- Marketing Lead Scoring: TP value = $120 (converted lead), FP cost = $8 (wasted sales call), FN cost = $25 (missed opportunity, low cost because leads are abundant), base rate = 8%.

import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix

np.random.seed(42)

# Simulate each scenario and find the optimal threshold.
# Your code here

a) Report the optimal threshold for each scenario.

b) Rank the three scenarios by how far the optimal threshold is from 0.50. Explain why the scenario with the largest gap has the most extreme threshold.

c) For Scenario B, compute how much money is lost per month by using a threshold of 0.50 instead of the optimal threshold. State this as both a dollar amount and a percentage of maximum achievable value.

d) Write a one-sentence business recommendation for each scenario that a non-technical manager could understand.

Exercise 3: Stakeholder Communication (Writing)

You are a data scientist at a regional hospital. Your 30-day readmission prediction model has the following performance:

• AUC: 0.79
• Precision at chosen threshold: 0.52
• Recall at chosen threshold: 0.71
• 1,200 discharges per month
• 15% readmission rate
• Follow-up program cost: $350 per patient
• Average readmission cost: $8,200

a) Write a one-paragraph executive summary (4--6 sentences) for the hospital's chief medical officer. Use zero technical jargon. Do not mention AUC, precision, or recall by name. Translate everything into clinical and financial terms.

b) The CMO asks: "What percentage of readmissions will we still miss?" Write a 2--3 sentence honest answer that acknowledges the limitation without undermining confidence in the model.

c) A nurse manager says: "I do not trust a computer to tell me which patients are at risk. I have been doing this for 20 years." Write a 3--4 sentence response that is respectful, acknowledges her expertise, and positions the model as a complement to clinical judgment.

d) The CFO asks: "What is the payback period?" Calculate the monthly net value of the model (including follow-up costs for both TPs and FPs) and determine how many months until the program costs are recovered. Assume a one-time implementation cost of $45,000 for the initial model build and infrastructure.

Exercise 4: Data Maturity Assessment (Analysis)

A mid-size e-commerce company (500 employees, $80M revenue) has the following characteristics:

• Customer data is in a Snowflake warehouse (centralized).
• The marketing team runs weekly reports from Looker dashboards.
• There is no formal data quality process; analysts manually check data when something "looks wrong."
• Feature engineering is done ad hoc in Jupyter notebooks; there is no feature store.
• No ML models are in production. The closest thing is a rule-based email segmentation system.
• The company has never run a formal A/B test. Product decisions are based on executive intuition and customer complaints.
• There is no CI/CD pipeline for data or models.
• There is no model governance framework.

a) Using the assess_data_maturity function from the chapter, score the company. What maturity level does it fall into?

b) The CEO says: "I want us to have a recommendation engine like Amazon within six months." Write a diplomatic 3--4 sentence response explaining why this timeline is unrealistic, what needs to happen first, and a more realistic first project.

c) Propose a data science roadmap with three phases: - Phase 1 (months 1--3): Foundation building - Phase 2 (months 4--6): First ML project - Phase 3 (months 7--12): Scaling

For each phase, list 2--3 specific deliverables and one success metric.

d) The CEO pushes back: "Our competitor launched an AI chatbot last month. We are falling behind." How do you respond? (Hint: revisit the "Five Responses to 'We Need AI'" framework from the chapter.)

Exercise 5: ROI Sensitivity Analysis (Code)

The ROI calculation depends on assumptions. Assumptions are often wrong. Build a sensitivity analysis for the StreamFlow churn model.

import numpy as np
import pandas as pd

np.random.seed(42)

# Base case assumptions
base_case = {
    "tp_value": 78.0,
    "fp_cost": 30.0,
    "fn_cost": 180.0,
    "intervention_success_rate": 0.60,
    "annual_subscriber_value": 180.0,
    "churn_rate": 0.12,
    "infrastructure_cost": 2500.0,
    "team_cost": 4000.0,
}

# Your task: vary each assumption by +/- 25% (one at a time)
# and compute the resulting monthly ROI.

a) Create a tornado chart (or a sorted bar chart) showing which assumption has the largest impact on ROI. Which single assumption matters most?

b) The VP of Finance says: "I do not believe the intervention success rate is 60%. I think it is closer to 35%." Recompute the ROI with a 35% success rate. Is the model still worth deploying?

c) At what intervention success rate does the model become unprofitable (negative ROI after operational costs)?

d) Write a one-paragraph risk summary for the finance team that explains: (1) the base case ROI, (2) the most sensitive assumption, and (3) the break-even point for the most sensitive assumption.

Exercise 6: The Hard Conversation Simulation (Writing + Analysis)

You are the lead data scientist at StreamFlow. The VP of Product wants to add a "recommended shows" feature to the home screen. The product team has been building it for three months. You run an A/B test for two weeks with 40,000 users per group:

• Control group conversion rate (clicks on a show): 12.4%
• Treatment group conversion rate: 12.8%
• p-value: 0.18
• 95% confidence interval for the effect: [-0.1%, +0.9%]

The VP says: "The trend is clearly positive. Let us launch. We have been working on this for months."

a) Is the result statistically significant at the 0.05 level? What does the confidence interval tell you about the plausible range of the true effect?

b) Compute the minimum detectable effect (MDE) for this test configuration (40,000 per group, alpha = 0.05, power = 0.80, base rate = 12.4%). Was the test adequately powered to detect a 0.4 percentage point lift?

from scipy import stats
import numpy as np

# Use the formula for MDE in a two-proportion z-test.
# MDE = (z_alpha/2 + z_beta) * sqrt(2 * p * (1-p) / n)
np.random.seed(42)

# Your code here

c) Write a 4--5 sentence response to the VP that: - Acknowledges the team's effort - Explains the result honestly - Proposes a concrete next step (extend the test, launch to a subset, etc.) - Does not use the words "statistically significant" (translate to business language)

d) The VP overrules you and launches anyway. Write a 2--3 sentence documentation note for the decision log. Include the date, the decision, the rationale, and what monitoring you will put in place.

Exercise 7: Portfolio Project Critique (Analysis)

Review the following portfolio project description from a hypothetical junior data scientist's GitHub:

Project: Customer Churn Prediction Used Kaggle Telco churn dataset. Performed EDA with pandas and seaborn. Trained logistic regression, random forest, and XGBoost. XGBoost achieved best accuracy (81%). Feature importance shows contract type and tenure are top predictors. Deployed with Streamlit.

a) List three specific weaknesses of this portfolio project from a hiring manager's perspective.

b) Rewrite the project description in 4--5 sentences that demonstrate business thinking. Use the guidance from this chapter: start with a business question, include an ROI estimate, mention limitations, and describe a decision recommendation.

c) What three additional artifacts would make this project stand out? (Hint: think about what this chapter teaches.)

d) A hiring manager asks in an interview: "Walk me through a time you had to convince a non-technical stakeholder to trust your model." Draft a 4--5 sentence answer using the StreamFlow or hospital readmission scenario from this chapter as if it were your own experience.

These exercises cover Chapter 34: The Business of Data Science. Return to the chapter for reference.