Chapter 25 Exercises: What Is a Model? Prediction, Explanation, and the Bias-Variance Tradeoff

How to use these exercises: Part A tests your understanding of core modeling concepts — what models are, why they simplify, and the difference between prediction and explanation. Part B applies the concepts to real-world scenarios. Part C involves Python code for train-test splitting and baselines. Part D pushes toward deeper analysis of the bias-variance tradeoff and ethical considerations.

Difficulty key: ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Advanced | ⭐⭐⭐⭐ Extension

Part A: Conceptual Understanding

Exercise 25.1 — Models as simplifications ⭐

For each of the following, explain what a model of this system would need to include and what it could safely ignore. Then explain why the simplification is useful.

1. A model predicting how long a pizza delivery will take
2. A model predicting whether a student will graduate on time
3. A model predicting tomorrow's temperature

Exercise 25.2 — Prediction vs. explanation ⭐

Classify each scenario as primarily a prediction task or an explanation task. Justify your answer.

1. A bank wants to know which loan applicants will default so it can reject risky applications.
2. A public health researcher wants to know whether a new vaccine reduces infection rates.
3. A streaming service wants to recommend movies users will enjoy.
4. An economist wants to understand why some countries grow faster than others.
5. A retailer wants to forecast next month's sales to plan inventory.

Exercise 25.3 — Supervised vs. unsupervised ⭐

For each task, state whether it is supervised or unsupervised learning. Identify the features and target if supervised, or describe the structure being sought if unsupervised.

1. Predicting house prices based on square footage and location
2. Grouping customers into segments based on purchasing behavior
3. Classifying emails as spam or not spam based on email content
4. Reducing a dataset with 50 variables down to 5 key components
5. Predicting whether a patient has diabetes based on blood test results

Exercise 25.4 — Identifying features and targets ⭐

For each modeling problem, list at least 4 plausible features and identify the target variable. Also state whether the problem is regression or classification.

1. Predicting the price of a used car
2. Predicting whether a flight will be delayed
3. Predicting a student's final exam score
4. Predicting whether a tumor is malignant or benign

Exercise 25.5 — The cardinal rule ⭐

A friend says: "I trained my model on all the data and then tested it on the same data. It got 99% accuracy!" Explain to your friend, in plain language, why this evaluation is unreliable. Use an analogy to make the point clear.

Exercise 25.6 — Overfitting vs. underfitting ⭐

A data scientist trains three models on the same dataset. Here are the results:

1. Which model is overfitting? How can you tell?
2. Which model is underfitting? How can you tell?
3. Which model has the best generalization? Why?

Exercise 25.7 — The bias-variance tradeoff in everyday life ⭐⭐

For each analogy, explain whether the situation reflects high bias, high variance, or a good balance:

1. A doctor who diagnoses every patient with the flu, regardless of symptoms
2. A chef who follows a completely different recipe every time someone orders the same dish
3. A weather app that gives reasonably accurate forecasts that are occasionally off by a few degrees
4. A student who memorizes practice exams word-for-word but can't answer questions phrased differently

Exercise 25.8 — Baseline thinking ⭐⭐

For each problem, define an appropriate baseline model and explain what it would predict:

1. Predicting whether a coin flip will be heads or tails (50/50 coin)
2. Predicting tomorrow's stock price
3. Predicting whether a passenger survived the Titanic (where ~38% survived)
4. Predicting a country's vaccination rate (mean rate = 82%)

Exercise 25.9 — The Goldilocks zone ⭐⭐

Explain, using the "connecting dots" analogy from the chapter, what happens at each stage:

1. You draw a horizontal flat line through a scatter plot that clearly shows an upward trend
2. You draw a straight line that captures the upward trend but misses some curvature
3. You draw a smooth curve that captures the overall shape of the data
4. You draw a jagged line that passes through every single point

Which of these represents underfitting? Overfitting? A good fit? Which would you expect to perform best on new data?

Part B: Applied Scenarios ⭐⭐

Exercise 25.10 — Framing a prediction problem

You work for a hospital and have a dataset with the following columns: patient_age, blood_pressure, cholesterol, exercise_hours_per_week, smoker (yes/no), and heart_disease (yes/no).

1. Identify the target and features for predicting heart disease
2. Is this regression or classification?
3. What would a good baseline model be?
4. What test_size would you choose and why?
5. What ethical concerns should you consider?

Exercise 25.11 — When models mislead ⭐⭐

A company builds a model to predict employee performance ratings based on features like years of experience, education level, and department. The model achieves 87% accuracy on the test set.

1. What could go wrong if this model is used for hiring or promotion decisions?
2. What biases might be in the training data?
3. Would you classify this as a prediction task or an explanation task? Why does that distinction matter here?
4. What baseline would you compare the 87% against?

Exercise 25.12 — Choosing features wisely ⭐⭐

You're building a model to predict whether a student will pass a course. You have access to many variables. For each potential feature below, discuss whether you should include it and why:

1. Number of assignments submitted
2. Student's name
3. Student's ZIP code
4. Midterm exam score
5. Whether the student passed a different course last semester
6. Student's ID number

Exercise 25.13 — The bias-variance tradeoff in practice ⭐⭐

A team is building a model to predict housing prices. They try three approaches:

• Approach A: Use only square footage (1 feature)
• Approach B: Use square footage, bedrooms, bathrooms, and neighborhood (4 features)
• Approach C: Use 200 features including carpet color, mailbox style, distance to nearest fire hydrant, etc.

Predict the likely bias and variance characteristics of each approach. Which is most likely to overfit? Underfit? Achieve the best generalization?

Exercise 25.14 — Interpreting train-test results ⭐⭐

Five models are evaluated on the same dataset. Analyze each result:

1. Fill in the "Verdict" column (underfitting, good fit, overfitting)
2. Which model would you deploy? Why?
3. Why might Model 5 not always be the best choice, even though it has the best test score?

Part C: Coding Exercises ⭐⭐

Exercise 25.15 — Train-test split in practice

Using the code below as a starting point, perform a train-test split and calculate the baseline MAE for predicting exam scores:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# Simulated student data
np.random.seed(42)
n = 200
data = pd.DataFrame({
    'study_hours': np.random.uniform(1, 15, n),
    'attendance_pct': np.random.uniform(50, 100, n),
    'previous_gpa': np.random.uniform(1.5, 4.0, n),
    'exam_score': None  # You'll generate this
})

# Generate exam_score based on features (with noise)
data['exam_score'] = (
    3 * data['study_hours'] +
    0.4 * data['attendance_pct'] +
    8 * data['previous_gpa'] +
    np.random.normal(0, 5, n)
)

Tasks: 1. Define X (features) and y (target) 2. Split into 80/20 training/test sets with random_state=42 3. Calculate the mean of y_train (the baseline prediction) 4. Calculate the baseline MAE on the test set 5. Print a summary of your results

X = data[['study_hours', 'attendance_pct', 'previous_gpa']]
y = data['exam_score']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

baseline_pred = y_train.mean()
baseline_mae = np.abs(y_test - baseline_pred).mean()

print(f"Training samples: {len(X_train)}")
print(f"Test samples: {len(X_test)}")
print(f"Baseline prediction (mean): {baseline_pred:.2f}")
print(f"Baseline MAE: {baseline_mae:.2f}")

Exercise 25.16 — Visualizing overfitting

Create a visualization that demonstrates overfitting using polynomial regression. Generate noisy data from a simple function, then fit polynomials of degrees 1, 3, and 12. Plot all three fits on the same or side-by-side plots.

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(0)
X = np.sort(np.random.uniform(0, 6, 25))
y = np.sin(X) + np.random.normal(0, 0.3, 25)

Tasks: 1. Fit polynomial models of degree 1, 3, and 12 2. Plot each fit alongside the data points 3. Add titles indicating "Underfitting," "Good Fit," and "Overfitting" 4. Which model would you expect to perform best on new data?

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
X_plot = np.linspace(0, 6, 200)

for ax, degree, label in zip(axes, [1, 3, 12],
    ['Underfitting (deg 1)', 'Good Fit (deg 3)', 'Overfitting (deg 12)']):
    coeffs = np.polyfit(X, y, degree)
    y_plot = np.polyval(coeffs, X_plot)

    ax.scatter(X, y, color='steelblue', zorder=5)
    ax.plot(X_plot, y_plot, color='coral', linewidth=2)
    ax.set_title(label)
    ax.set_ylim(-2, 2.5)

plt.tight_layout()
plt.show()

Exercise 25.17 — Comparing train and test scores ⭐⭐

Using the student exam data from Exercise 25.15, fit polynomial regression models of increasing complexity and track both training and test scores to observe overfitting:

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

Tasks: 1. Use only study_hours as the feature (for visualization purposes) 2. Fit polynomial models of degrees 1, 2, 3, 5, 10, and 15 3. Record the R² score on both training and test data for each degree 4. Plot training R² and test R² vs. polynomial degree 5. At what degree does overfitting become apparent?

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

X_1d = data[['study_hours']]
X_tr, X_te, y_tr, y_te = train_test_split(
    X_1d, y, test_size=0.2, random_state=42)

degrees = [1, 2, 3, 5, 10, 15]
train_scores, test_scores = [], []

for d in degrees:
    poly = PolynomialFeatures(d)
    X_tr_p = poly.fit_transform(X_tr)
    X_te_p = poly.transform(X_te)
    model = LinearRegression().fit(X_tr_p, y_tr)
    train_scores.append(model.score(X_tr_p, y_tr))
    test_scores.append(model.score(X_te_p, y_te))

plt.plot(degrees, train_scores, 'o-', label='Training')
plt.plot(degrees, test_scores, 's-', label='Test')
plt.xlabel('Polynomial Degree')
plt.ylabel('R² Score')
plt.title('Overfitting: Train vs Test')
plt.legend()
plt.show()

Exercise 25.18 — The effect of training set size ⭐⭐

Investigate how training set size affects model performance. Using the exam data:

1. Fix the test set (use the same 20% test split)
2. Train a LinearRegression model using increasing amounts of training data: 10%, 20%, 40%, 60%, 80%, and 100% of the training set
3. Record the test R² for each training set size
4. Plot test R² vs. training set size
5. What pattern do you observe? How does this relate to the bias-variance tradeoff?

from sklearn.linear_model import LinearRegression

fractions = [0.1, 0.2, 0.4, 0.6, 0.8, 1.0]
test_r2 = []

for frac in fractions:
    n_samples = int(len(X_train) * frac)
    X_sub = X_train.iloc[:n_samples]
    y_sub = y_train.iloc[:n_samples]
    model = LinearRegression().fit(X_sub, y_sub)
    test_r2.append(model.score(X_test, y_test))

plt.plot(fractions, test_r2, 'o-')
plt.xlabel('Fraction of Training Data Used')
plt.ylabel('Test R²')
plt.title('Learning Curve')
plt.show()

Exercise 25.19 — Baseline comparison ⭐⭐

Build a complete baseline analysis for the progressive project's vaccination data:

# Simulated country indicators
np.random.seed(42)
n = 150
countries = pd.DataFrame({
    'gdp_per_capita': np.random.lognormal(9, 1.2, n),
    'health_spending_pct': np.random.uniform(2, 12, n),
    'education_index': np.random.uniform(0.3, 0.95, n),
    'vaccination_rate': None
})

countries['vaccination_rate'] = (
    10 * np.log(countries['gdp_per_capita'] / 1000) +
    2 * countries['health_spending_pct'] +
    30 * countries['education_index'] +
    np.random.normal(0, 8, n)
).clip(20, 100)

Tasks: 1. Define features and target 2. Perform an 80/20 train-test split 3. Calculate the mean and median baselines 4. Compute MAE for both baselines on the test set 5. Which baseline is better? Why might the median baseline outperform the mean?

X = countries[['gdp_per_capita', 'health_spending_pct',
               'education_index']]
y = countries['vaccination_rate']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

mean_baseline = y_train.mean()
median_baseline = y_train.median()

mae_mean = np.abs(y_test - mean_baseline).mean()
mae_median = np.abs(y_test - median_baseline).mean()

print(f"Mean baseline MAE: {mae_mean:.2f}")
print(f"Median baseline MAE: {mae_median:.2f}")

Exercise 25.20 — Random state exploration ⭐⭐

The random_state parameter in train_test_split controls the random split. Investigate how different splits affect your baseline results:

1. Run train_test_split with random_state values 0 through 9
2. For each split, compute the baseline MAE (mean baseline)
3. Calculate the mean and standard deviation of the 10 MAE values
4. What does the standard deviation tell you about the reliability of a single train-test split?

maes = []
for rs in range(10):
    _, X_te, _, y_te = train_test_split(
        X, y, test_size=0.2, random_state=rs)
    baseline = y_train.mean()
    maes.append(np.abs(y_te - baseline).mean())

print(f"MAE across splits: {np.mean(maes):.2f} +/- {np.std(maes):.2f}")

Part D: Synthesis and Critical Thinking ⭐⭐⭐

Exercise 25.21 — The model simplification paradox

George Box said "All models are wrong, but some are useful." Yet we also say "simple models often beat complex ones." Isn't this contradictory? If all models are wrong, shouldn't we try to make them as accurate (complex) as possible?

Write a paragraph explaining why the answer is no, using the bias-variance tradeoff to support your argument.

Exercise 25.22 — Ethical analysis: predictive policing ⭐⭐⭐

A city considers using a machine learning model to predict where crimes will occur, deploying more police to high-prediction areas.

1. What target variable and features might such a model use?
2. What biases could exist in the training data (historical crime data)?
3. How might the model create a feedback loop?
4. Using the prediction vs. explanation framework, why is explanation especially important here?
5. What safeguards would you recommend?

Exercise 25.23 — The "just predict the average" challenge ⭐⭐⭐

Many real-world machine learning models barely beat the baseline. Research suggests that for some prediction tasks (stock prices, political elections, economic forecasts), simple baselines perform nearly as well as complex models.

1. Why might this be the case?
2. Does this mean machine learning is useless for these problems?
3. What does it tell us about the "signal-to-noise ratio" in these domains?
4. How should a data scientist communicate this finding to stakeholders who expected a breakthrough?

Exercise 25.24 — Design your own modeling problem ⭐⭐⭐

Choose a topic that interests you (sports, music, health, environment, etc.) and design a complete supervised learning problem:

1. Define the question in plain language
2. Identify the target variable (regression or classification?)
3. List 5-8 plausible features
4. Describe what training data you would need and where you might find it
5. Define an appropriate baseline model
6. Describe potential ethical concerns
7. Predict whether this will be a high-signal or low-signal problem and explain why

Exercise 25.25 — Reflection: what changes now? ⭐⭐⭐⭐

Looking back at Chapters 1 through 24, you were learning to understand data that exists. Starting now, you're learning to predict data that doesn't exist yet.

Write a reflection (3-5 paragraphs) on:

1. How the shift from description to prediction changes what "success" means in data analysis
2. Why the concepts from earlier chapters (data cleaning, visualization, statistical thinking) are still essential for modeling
3. One real-world prediction problem you would like to solve and how you would frame it using the vocabulary from this chapter
4. What concerns you most about using models to make predictions about people