Chapter 6 Exercises: The Business of Machine Learning

Section A: Problem Framing

Exercise 6.1 — Translating Business Problems

For each of the following business statements, rewrite them as a precise ML problem statement that specifies: (a) the prediction target, (b) the prediction type (classification, regression, ranking, or clustering), (c) the prediction window, and (d) the decision or action the prediction will inform.

a) "We want to reduce customer churn." b) "We need to forecast how many units of each product to stock next month." c) "We should figure out which customers are most similar to each other." d) "Our customer service team is overwhelmed — can AI help route tickets?" e) "We lose money on returns. Can we predict which orders will be returned?"

Exercise 6.2 — Evaluating ML Suitability

A regional hospital chain is considering five potential ML projects. For each, evaluate whether ML is appropriate using the criteria from Section 6.2 (patterns exist, patterns are learnable, decisions have value, errors are tolerable, problem is too complex for rules, labeled data exists). If ML is not appropriate, explain why and suggest an alternative approach.

a) Predicting which patients will miss their scheduled appointments (no-shows) b) Predicting the exact date a specific patient will die c) Predicting whether an insurance claim is likely fraudulent d) Automatically generating the hospital's annual budget e) Predicting which surgical equipment will need maintenance in the next 30 days

Exercise 6.3 — The "Predict Everything" Trap

A marketing VP approaches you and says: "We want a model that predicts which customers will buy, what they'll buy, when they'll buy it, how much they'll spend, and through which channel." Write a two-paragraph response that (a) explains why this is problematic and (b) proposes a better-scoped alternative as a starting point.

Exercise 6.4 — Problem Framing Gone Wrong

Read the following scenario and identify the problem framing error:

A bank builds a credit risk model to predict whether loan applicants will default within 24 months. The model achieves strong performance on test data. After deployment, the bank notices that the model rarely approves loans for applicants under 25 years old, because younger borrowers historically default at higher rates. The bank's lending in the 18-25 age demographic drops by 70 percent.

What business objective was the model actually optimized for? What business objective should it have been optimized for? How would you reframe the problem?

Section B: The ML Canvas

Exercise 6.5 — Complete ML Canvas

You are the Head of Data Science at an online education platform (100,000 active students). Student completion rates for courses have dropped from 34 percent to 22 percent over the past two years. The CEO wants to improve completion rates.

Complete a full ML Canvas (all 10 sections from Section 6.3) for a model designed to address this problem. Be specific about data sources, features, evaluation metrics, and failure modes.

Exercise 6.6 — Comparative ML Canvas

A logistics company is considering two ML projects: - Project A: Predicting delivery delays before they occur - Project B: Optimizing truck loading patterns to maximize vehicle utilization

Create a simplified ML Canvas (Value Proposition, Prediction Target, Data Sources, Evaluation Metrics, and Failure Modes) for each project. Then write a one-paragraph recommendation for which project should be prioritized, supported by your analysis.

Exercise 6.7 — Spotting Canvas Gaps

Review the following incomplete ML Canvas and identify three critical gaps or flaws that would likely cause project failure:

Section C: Metrics and Business Value

Exercise 6.8 — Precision vs. Recall Tradeoffs

For each of the following scenarios, determine whether the business should optimize for precision or recall, and explain your reasoning:

a) An e-commerce fraud detection system where the average fraudulent transaction is $120 and manual review of a flagged transaction costs $3 b) A manufacturing quality control system where defective products that reach customers result in an average warranty claim of $800, and pulling a non-defective product for additional inspection costs $12 c) A job applicant screening system where interviewing a candidate costs the company $500 in employee time, and missing a great hire costs an estimated $50,000 in lost productivity d) An email spam filter for a CEO's inbox where missing an important email could cost a major business deal, and reviewing a spam email takes 5 seconds

Exercise 6.9 — Connecting Models to Dollars

An insurance company's claims fraud detection model has the following performance on a test set of 10,000 claims:

• True Positives: 180 (correctly identified fraudulent claims)
• False Positives: 420 (legitimate claims flagged as fraud)
• False Negatives: 120 (fraudulent claims missed)
• True Negatives: 9,280 (legitimate claims correctly passed)

Given: - Average cost of investigating a flagged claim: $200 - Average value of a caught fraudulent claim: $8,000 - Average cost of a missed fraudulent claim: $8,000

Calculate: (a) the net value of the model per 10,000 claims, (b) the break-even precision (the minimum precision at which the model creates positive value), and (c) how the net value would change if the investigation cost increased to $500.

Exercise 6.10 — The Accuracy Illusion

A model predicts whether customers will respond to a direct mail campaign. The dataset contains 50,000 customers, of whom 2 percent (1,000) have historically responded.

a) What accuracy would a "predict no response for everyone" model achieve? b) If the company sends 5,000 mailers at $2 each and each positive response generates $100 in revenue, what is the expected profit using random targeting (no model)? c) Suppose the model identifies 3,000 customers to target, of whom 600 actually respond. What is the model's precision? What is its recall? What is the expected profit? d) Is accuracy a useful metric for evaluating this model? Why or why not?

Section D: Failure Modes

Exercise 6.11 — Data Leakage Detection

For each of the following scenarios, determine whether data leakage is present. If so, identify the source of the leak and explain how to fix it.

a) A hospital readmission model uses "number of medications prescribed at discharge" as a feature to predict whether a patient will be readmitted within 30 days. b) A customer churn model uses "customer satisfaction survey score" as a feature, where the survey was administered the same week the customer canceled their account. c) A credit default model uses the applicant's credit score, income, debt-to-income ratio, and employment length as features to predict loan default. d) A product return prediction model uses the customer's average product rating (across all products they've ever bought, including the product being predicted) as a feature.

Exercise 6.12 — Failure Mode Autopsy

Read the following scenario and identify which failure mode(s) from Section 6.5 are at play. Then write a brief (3-5 sentence) recommendation for what should have been done differently.

A retail bank spends $1.5 million building a credit scoring model using a team of four data scientists over nine months. The model outperforms the bank's existing scorecard on all statistical metrics. When the model is presented to the credit risk team for deployment, they raise multiple concerns: the model uses features that aren't available at the time of the lending decision, the model's predictions need to be integrated into the loan origination system (which has a six-month IT development backlog), and the bank's regulators require model explainability that the chosen algorithm (a deep neural network) cannot easily provide.

Exercise 6.13 — Scope Creep Simulation

You are managing an ML project to predict equipment failure in a manufacturing plant. Your original scope: predict which machines will fail in the next 7 days so that maintenance can be scheduled proactively.

The following requests come in over the course of the project. For each, decide whether to accept it (within scope), defer it (separate project), or reject it (not appropriate for ML). Justify each decision.

a) "Can the model also predict what part will fail, not just which machine?" b) "Can we add data from our European plants, even though they use different equipment?" c) "Can the model also estimate how much money we save by doing preventive maintenance versus reactive maintenance?" d) "We'd like the model to automatically order replacement parts when it predicts a failure." e) "Can we make the model explainable so technicians understand why a machine is flagged?"

Section E: Build vs. Buy

Exercise 6.14 — Build vs. Buy Analysis

For each of the following scenarios, evaluate the five dimensions of the build-vs-buy framework (strategic differentiation, data uniqueness, talent availability, time pressure, TCO at scale) and provide a recommendation.

a) A mid-size law firm wants to automate the review of contracts for standard clauses. They have no data scientists on staff. b) A major ride-sharing company wants to build a dynamic pricing model that responds to real-time supply and demand. c) A hospital system wants to implement speech-to-text for clinical documentation. d) A hedge fund wants to build a sentiment analysis system that processes financial news and social media to inform trading decisions.

Exercise 6.15 — Vendor Evaluation

You are evaluating three vendor solutions for customer churn prediction. Create an evaluation scorecard with at least eight criteria (drawn from the chapter or your own experience). For each criterion, define what a "strong," "moderate," and "weak" score looks like.

Exercise 6.16 — The Hybrid Approach

A mid-size e-commerce company ($200M revenue, 15 employees in the data team) is deciding between: - Option A: Build everything on AWS SageMaker (build-on-buy) - Option B: Purchase a specialized e-commerce ML platform (full buy) - Option C: Build a custom ML infrastructure from scratch (full build)

Write a one-page memo recommending the best option. Address cost, time-to-value, customization, scalability, and talent requirements.

Section F: Team Composition and Planning

Exercise 6.17 — Team Design

Design the minimum viable team for each of the following ML projects. For each role, specify whether it's full-time, part-time, or advisory. Justify any roles you include or exclude.

a) A startup (50 employees) building its first recommendation engine for a mobile app b) A Fortune 500 bank deploying a new fraud detection model across all branches c) A government health agency building a disease surveillance model

Exercise 6.18 — The Unicorn Problem

A hiring manager shows you the following job description:

Senior Data Scientist Requirements: PhD in Machine Learning or Statistics. 7+ years of experience. Expert in Python, R, SQL, Spark, TensorFlow, PyTorch, and scikit-learn. Experience deploying models to production on AWS and GCP. Strong product sense. Excellent communication skills. Domain expertise in healthcare preferred. Salary: $140,000-$160,000.

Identify three problems with this job description. Propose how the hiring manager should restructure the role (or split it into multiple roles) to be more realistic and effective.

Exercise 6.19 — Estimation Challenge

You are asked to estimate the timeline for the following ML project: predicting customer lifetime value (CLV) for a subscription-based SaaS company with 50,000 customers and 3 years of transaction history.

a) Outline the phases of the project using the framework from Section 6.8. b) Estimate the duration of each phase, providing a range (optimistic, expected, pessimistic). c) Identify the three largest sources of uncertainty and explain how you would reduce them. d) Draft a two-sentence communication to the executive sponsor explaining the timeline uncertainty.

Section G: Governance and Economics

Exercise 6.20 — Stage Gate Design

Design a stage-gate process for ML projects at a mid-size financial services firm. For each gate: (a) define the gate criteria (what must be demonstrated), (b) identify who should review, and (c) specify the possible outcomes (proceed, revise, stop). Consider the firm's regulatory obligations.

Exercise 6.21 — TCO Analysis

Estimate the 3-year total cost of ownership for a product recommendation engine at a mid-size e-commerce company ($500M annual revenue, 5M customers). Use the cost categories from Section 6.11. State your assumptions clearly. Then calculate the minimum annual revenue increase needed to justify the investment.

Exercise 6.22 — The POC Trap

Your data science team has built a proof of concept for a dynamic pricing model. The POC uses a static dataset, runs in a Jupyter notebook, and takes 3 hours to generate prices for 10,000 SKUs. The CMO is excited: "Ship it by next month."

Write a response (as if to the CMO) that (a) validates the POC's success, (b) explains the gap between POC and production, (c) provides a realistic timeline and resource estimate for production deployment, and (d) proposes interim steps that deliver value before the production system is ready.

Section H: Integrative Exercises

Exercise 6.23 — Full Project Scoping

Choose one of the following business scenarios and produce a complete project scope document that includes: - Problem statement (one sentence) - ML Canvas (all 10 sections) - Success metrics (model metrics AND business metrics) - Team composition (roles and time allocation) - Timeline estimate (with phases and uncertainty ranges) - Top 3 risks and mitigation strategies - Build vs. buy recommendation with justification

Scenarios: a) An airline wants to predict flight delays to proactively rebook passengers b) A pharmaceutical company wants to predict which clinical trials will succeed c) A real estate platform wants to predict property values more accurately than current appraisals d) A university wants to predict which admitted students will actually enroll

Exercise 6.24 — Prioritization Matrix

You are the VP of Data Science at a health insurance company. Your team has identified eight potential ML use cases:

1. Claims fraud detection
2. Member churn prediction
3. Provider network optimization
4. Personalized wellness recommendations
5. Prior authorization automation
6. Call center volume forecasting
7. Drug interaction prediction
8. Premium pricing optimization

Score each use case on Business Impact (1-5), Technical Feasibility (1-5), and Data Readiness (1-5). Present your scores in a matrix, identify the top 3 priorities, and explain your rationale. Note: there is no single correct answer — the exercise is about the reasoning process.

Exercise 6.25 — The Five Questions Applied

Apply Professor Okonkwo's Five Questions to each of the following ML project proposals. For each question, provide a "green" (ready), "yellow" (addressable risk), or "red" (showstopper) rating, with a brief explanation.

a) A city government wants to use ML to predict which buildings are at risk of fire code violations and prioritize inspections accordingly. b) A social media startup (6 months old, 10,000 users) wants to build a personalized content feed algorithm. c) A large retailer wants to predict which products will go viral on social media and stock up in advance.

Suggested approach: Work through these exercises in order within each section, but feel free to jump between sections based on your interests. Exercises marked with a scenario are best completed in groups of 2-4, simulating the cross-functional teams described in this chapter.