Chapter 29 Exercises: Evaluating Models

How to use these exercises: Start with the conceptual questions to make sure you understand why different metrics exist. Then move to applied exercises where you compute and interpret these metrics in code. The real-world and synthesis sections push you to think about which metrics matter in specific contexts — the skill that separates beginners from practitioners.

Difficulty key: ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Advanced | ⭐⭐⭐⭐ Extension

Part A: Conceptual Understanding ⭐

Exercise 29.1 — The accuracy trap

A model for predicting whether a customer will churn achieves 92% accuracy on a dataset where 92% of customers do NOT churn. Without running any code, answer:

1. What is the simplest model that achieves 92% accuracy on this dataset?
2. What is the recall of this simple model for the "churn" class?
3. Why is this model useless despite its high accuracy?

Exercise 29.2 — Confusion matrix from scratch

A model makes the following predictions for 20 patients tested for a disease:

Construct the confusion matrix. Then compute accuracy, precision, recall, and F1 score by hand.

         Predicted+  Predicted-
Actual+      5           3
Actual-      2          10

Exercise 29.3 — Precision vs. recall scenarios

For each of the following scenarios, state whether you would prioritize precision or recall, and explain why:

1. A model that flags potentially defective airplane engine parts for manual inspection
2. A model that recommends movies to Netflix users
3. A model that identifies potential spam emails for the spam folder
4. A model that screens blood donations for HIV
5. A model that identifies candidates for a marketing promotion

Exercise 29.4 — F1 score intuition

Why does the F1 score use the harmonic mean of precision and recall instead of the arithmetic mean? Demonstrate with an example where precision = 1.0 and recall = 0.01.

Exercise 29.5 — ROC curve reading

Describe what each of the following ROC curves would look like and what kind of model would produce each:

1. A curve that goes straight up to (0, 1) and then across to (1, 1)
2. A curve that follows the diagonal from (0, 0) to (1, 1)
3. A curve that bows toward the upper-left corner
4. A curve that bows toward the lower-right corner

Exercise 29.6 — Cross-validation purpose

Explain in your own words why 5-fold cross-validation gives a more reliable estimate of model performance than a single 80/20 train/test split. What specific problem does it address?

Exercise 29.7 — Regression metrics comparison

A model predicts house prices (in thousands of dollars) with the following actual vs. predicted values:

Compute MAE and RMSE by hand. Why is RMSE larger than MAE? What does that tell you about the errors?

Part B: Applied Exercises ⭐⭐

Exercise 29.8 — Confusion matrix in code

Load the Breast Cancer dataset from scikit-learn (from sklearn.datasets import load_breast_cancer). Train a logistic regression model with a 70/30 split. Generate and display the confusion matrix using ConfusionMatrixDisplay. How many false negatives are there? In medical terms, what does a false negative mean?

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.3, random_state=42
)
model = LogisticRegression(max_iter=5000)
model.fit(X_train, y_train)

Exercise 29.9 — Classification report interpretation

Using the same breast cancer model from Exercise 29.8, generate a classification_report. Write a one-paragraph interpretation for a non-technical audience (e.g., a hospital administrator) explaining: what the model does, how well it performs, and what its biggest weakness is.

Exercise 29.10 — Threshold tuning

Using the breast cancer logistic regression model, experiment with different classification thresholds. For thresholds of [0.3, 0.4, 0.5, 0.6, 0.7], compute the precision and recall for the malignant class. Plot precision and recall vs. threshold on the same chart. At which threshold would you deploy this model for medical screening? Why?

import numpy as np
y_proba = model.predict_proba(X_test)[:, 1]

thresholds = [0.3, 0.4, 0.5, 0.6, 0.7]
for t in thresholds:
    y_pred_t = (y_proba >= t).astype(int)
    # Compute precision and recall for each threshold

Exercise 29.11 — ROC curves for model comparison

Train three models on the breast cancer data (logistic regression, decision tree with max_depth=5, and random forest with 200 trees). Plot all three ROC curves on the same chart with their AUC values in the legend. Which model has the best AUC? Does the ranking match what you expected?

from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt

# For each model: fit, get predict_proba, compute roc_curve and AUC
# Plot all three on the same axes

Exercise 29.12 — Cross-validation comparison

Using the breast cancer dataset, compare all three models using 5-fold stratified cross-validation with four different scoring metrics: accuracy, precision, recall, and F1. Create a summary table showing mean and standard deviation for each model and metric. Which model would you choose, and does the choice depend on which metric you prioritize?

from sklearn.model_selection import cross_val_score, StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for metric in ['accuracy', 'precision', 'recall', 'f1']:
    for name, model in models.items():
        scores = cross_val_score(model, X, y, cv=skf, scoring=metric)
        print(f"{name} - {metric}: {scores.mean():.3f} +/- {scores.std():.3f}")

Exercise 29.13 — Learning curve diagnosis

Plot the learning curve for a decision tree with max_depth=15 on the breast cancer dataset. Is the model overfitting, underfitting, or well-fitted? Then plot the learning curve for a decision tree with max_depth=2. Compare the two curves and explain what each pattern tells you.

from sklearn.model_selection import learning_curve
import numpy as np

train_sizes, train_scores, test_scores = learning_curve(
    DecisionTreeClassifier(max_depth=15),
    X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10)
)

Exercise 29.14 — Regression evaluation

Load the California Housing dataset (from sklearn.datasets import fetch_california_housing). Train a random forest regressor. Compute MAE, RMSE, and R² on the test set. Write a sentence interpreting each metric for someone who wants to know "how good is this model at predicting house prices?"

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

housing = fetch_california_housing()
# Split, train, predict, evaluate

Part C: Real-World Applications ⭐⭐⭐

Exercise 29.15 — Elena's vaccination model evaluation

Elena built a model to predict which communities have low vaccination coverage. Her classification report shows:

              precision    recall  f1-score
Low coverage      0.72      0.85      0.78
High coverage     0.91      0.82      0.86

1. What type of error is most dangerous for Elena's use case?
2. Which class should she focus on improving?
3. If she could improve either precision or recall for the "Low coverage" class, which should she prioritize and why?

Exercise 29.16 — Marcus's sales prediction

Marcus built a regression model predicting daily bakery sales. His model has: - MAE = $145 - RMSE = $312 - R² = 0.64

Interpret these three numbers in plain English for Marcus (who doesn't know statistics). Also: the big gap between MAE and RMSE suggests something about the model's errors. What is it, and what might Marcus want to investigate?

Exercise 29.17 — The metric mismatch

A data science team optimized their email spam filter for the highest F1 score and achieved F1 = 0.92. When they deployed it, users complained loudly. Investigating, they found the model had precision = 0.87 and recall = 0.97. Why were users unhappy despite the excellent F1 score? What metric should they have optimized instead?

Exercise 29.18 — Jordan's grading fairness

Jordan built a model predicting whether a student gets an A, using features like study_hours, attendance, prior_gpa, and professor_id. The overall accuracy is 0.79 and F1 is 0.72. But when Jordan checks performance by demographic group, they find:

• Group A: Recall = 0.82
• Group B: Recall = 0.61

What does this disparity suggest? Is the model "fair"? What should Jordan investigate next?

Exercise 29.19 — Threshold selection in practice

You've built a model that predicts whether a patient should be referred for additional diagnostic testing. Your hospital has limited testing capacity — they can handle about 200 referrals per month, but your model would flag 350 patients at the default threshold of 0.5. How would you use the ROC curve and threshold adjustment to solve this practical constraint? What trade-off are you making?

Part D: Synthesis and Critical Thinking ⭐⭐⭐

Exercise 29.20 — Metric selection matrix

Create a table with the following domains as rows: medical diagnosis, spam filtering, credit card fraud, movie recommendations, self-driving car obstacle detection, job applicant screening. For each domain, specify: (a) which metric matters most, (b) whether false positives or false negatives are more costly, and (c) why.

Exercise 29.21 — Cross-validation edge cases

Explain what would go wrong in each of the following scenarios:

1. Using regular k-fold (not stratified) cross-validation on a dataset with 2% positive class
2. Using 50-fold cross-validation on a dataset with 100 samples
3. Using 2-fold cross-validation when comparing two complex models
4. Computing cross-validation scores and then using those same scores to report final model performance

Exercise 29.22 — The full evaluation workflow

Write out, step by step, a complete evaluation workflow for a binary classification problem. Include: data splitting strategy, metric selection, model comparison, final reporting. For each step, explain why it's done that way.

Exercise 29.23 — Designing an evaluation plan

You're tasked with building a model that predicts which students are at risk of failing a course, so the university can offer tutoring. Design a complete evaluation plan: which metrics you'd use, how you'd validate the model, how you'd check for fairness across demographic groups, and how you'd communicate results to university administrators who don't know machine learning.

Part E: Extension Challenges ⭐⭐⭐⭐

Exercise 29.24 — Precision-recall curves

Research precision-recall (PR) curves as an alternative to ROC curves. Using the breast cancer dataset, plot the PR curve for a logistic regression and a random forest. When are PR curves more informative than ROC curves? (Hint: think about class imbalance.) Write a paragraph comparing the two approaches.

from sklearn.metrics import precision_recall_curve, average_precision_score

precision, recall, thresholds = precision_recall_curve(y_test, y_proba)
ap = average_precision_score(y_test, y_proba)

Exercise 29.25 — Custom scoring functions

Sometimes standard metrics don't capture the business cost of errors. Suppose each false negative costs $500 (missed at-risk student who fails) and each false positive costs $100 (unnecessary tutoring offered). Write a custom scoring function that computes the total cost of a model's predictions and use it with cross_val_score. Compare how this custom metric ranks your three models versus how F1 ranks them.

from sklearn.metrics import make_scorer, confusion_matrix

def total_cost(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    cost = fn * 500 + fp * 100
    return -cost  # Negative because sklearn maximizes scores

cost_scorer = make_scorer(total_cost)
scores = cross_val_score(model, X, y, cv=5, scoring=cost_scorer)