Chapter 30 Exercises: The Machine Learning Workflow

How to use these exercises: This is the capstone exercise set for Part V. The conceptual questions test your understanding of workflow design and data leakage. The applied exercises have you building complete pipelines from scratch. The real-world and synthesis sections ask you to design workflows for novel problems. Every code exercise should be done with pipelines — no more preprocessing outside the pipeline.

Difficulty key: ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Advanced | ⭐⭐⭐⭐ Extension

Part A: Conceptual Understanding ⭐

Exercise 30.1 — Spot the leakage

For each of the following workflows, identify whether data leakage occurs and explain how:

1. You use StandardScaler().fit_transform(X) on the full dataset, then split into train/test.
2. You split into train/test, then use StandardScaler().fit_transform(X_train) and StandardScaler().transform(X_test).
3. You compute the median of a feature across the full dataset to fill missing values, then split into train/test.
4. You split into train/test, then perform feature selection on the training set using SelectKBest, and then apply the same selected features to the test set.

Exercise 30.2 — Pipeline benefits

List four specific problems that scikit-learn Pipeline solves compared to manually chaining preprocessing and modeling steps. For each problem, give a concrete example of what could go wrong without a pipeline.

Exercise 30.3 — Parameter naming conventions

In a pipeline with this structure:

pipe = Pipeline([
    ('preprocessor', ColumnTransformer([
        ('num', StandardScaler(), numeric_features),
        ('cat', OneHotEncoder(), categorical_features)
    ])),
    ('model', RandomForestClassifier())
])

Write the full parameter names for: (a) the number of trees in the random forest, (b) the drop parameter of the OneHotEncoder. Explain the double-underscore convention.

Exercise 30.4 — Grid search math

You have a parameter grid with: - n_estimators: [100, 200, 300, 400] - max_depth: [3, 5, 8, 12, None] - min_samples_leaf: [1, 5, 10]

Using 5-fold cross-validation, how many total model fits will GridSearchCV perform? If each fit takes 2 seconds, how long will the search take? How would RandomizedSearchCV with n_iter=30 change this?

Exercise 30.5 — The test set rule

Explain the "sealed envelope" principle for test sets. Then describe what happens if you violate it — specifically, how the test set becomes a validation set and why your reported performance will be optimistically biased.

Exercise 30.6 — Reproducibility checklist

Create a checklist of at least 6 things you should do to make a machine learning analysis reproducible. For each item, explain what would go wrong if you skip it.

Part B: Applied Exercises ⭐⭐

Exercise 30.7 — Your first pipeline

Build a simple pipeline that chains StandardScaler and LogisticRegression. Train it on the Breast Cancer dataset and evaluate using 5-fold cross-validation. Then build the same model WITHOUT a pipeline (scaling manually) and verify that the cross-validation scores are slightly different (because of leakage in the manual approach).

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score

data = load_breast_cancer()
X, y = data.data, data.target

# With pipeline (correct)
pipe = Pipeline([('scaler', StandardScaler()),
                 ('model', LogisticRegression(max_iter=5000))])
scores_pipe = cross_val_score(pipe, X, y, cv=5, scoring='accuracy')

# Without pipeline (leaky)
X_scaled = StandardScaler().fit_transform(X)
lr = LogisticRegression(max_iter=5000)
scores_leaky = cross_val_score(lr, X_scaled, y, cv=5, scoring='accuracy')

print(f"Pipeline: {scores_pipe.mean():.4f}")
print(f"Leaky:    {scores_leaky.mean():.4f}")

Exercise 30.8 — ColumnTransformer practice

Create a ColumnTransformer that applies StandardScaler to ['sepal length (cm)', 'sepal width (cm)'] and OneHotEncoder to a manually created categorical column (add a random 'color' column with values 'red', 'blue', 'green' to the Iris dataset). Fit the transformer on training data and examine the output.

import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

iris = load_iris(as_frame=True)
df = iris.frame
df['color'] = np.random.choice(['red', 'blue', 'green'], size=len(df))

numeric_cols = ['sepal length (cm)', 'sepal width (cm)']
cat_cols = ['color']

ct = ColumnTransformer([
    ('num', StandardScaler(), numeric_cols),
    ('cat', OneHotEncoder(drop='first'), cat_cols)
])

X_transformed = ct.fit_transform(df[numeric_cols + cat_cols])
print(f"Original shape: {df[numeric_cols + cat_cols].shape}")
print(f"Transformed shape: {X_transformed.shape}")

Exercise 30.9 — GridSearchCV

Using the Breast Cancer dataset, build a pipeline with StandardScaler and RandomForestClassifier. Use GridSearchCV to search over n_estimators [50, 100, 200] and max_depth [3, 5, 8, None]. Report the best parameters and the best cross-validated F1 score.

from sklearn.model_selection import GridSearchCV

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier(random_state=42))
])

param_grid = {
    'model__n_estimators': [50, 100, 200],
    'model__max_depth': [3, 5, 8, None]
}

gs = GridSearchCV(pipe, param_grid, cv=5, scoring='f1', n_jobs=-1)
gs.fit(X_train, y_train)
print(f"Best params: {gs.best_params_}")
print(f"Best F1: {gs.best_score_:.3f}")

Exercise 30.10 — Model comparison pipeline

Build three separate pipelines (logistic regression, decision tree, random forest) for the Breast Cancer dataset. Run GridSearchCV for each with appropriate parameter grids. Compare the best cross-validated F1 scores and select the overall best model. Evaluate the winner on the test set.

Exercise 30.11 — RandomizedSearchCV

Repeat Exercise 30.9 using RandomizedSearchCV instead of GridSearchCV, with n_iter=15 and parameter distributions from scipy.stats. Compare the best found parameters and score to the GridSearchCV results. Which approach was faster? Did they find similar hyperparameters?

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'model__n_estimators': randint(50, 300),
    'model__max_depth': [3, 5, 8, 12, None],
    'model__min_samples_leaf': randint(1, 20)
}

rs = RandomizedSearchCV(pipe, param_dist, n_iter=15, cv=5,
                         scoring='f1', random_state=42, n_jobs=-1)
rs.fit(X_train, y_train)

Exercise 30.12 — Save and load

Train a pipeline on the Breast Cancer dataset, save it using joblib, load it back, and verify that predictions from the loaded model match the original. Print the first 10 predictions from each to confirm they're identical.

import joblib

pipe.fit(X_train, y_train)
original_preds = pipe.predict(X_test)

joblib.dump(pipe, 'breast_cancer_pipe.joblib')
loaded_pipe = joblib.load('breast_cancer_pipe.joblib')
loaded_preds = loaded_pipe.predict(X_test)

print(f"Predictions match: {(original_preds == loaded_preds).all()}")

Exercise 30.13 — Missing value handling in pipelines

The Breast Cancer dataset has no missing values, but real data does. Create a pipeline that includes SimpleImputer (from sklearn.impute) to handle missing values before scaling. Test it by manually introducing NaN values into 5% of the training data.

from sklearn.impute import SimpleImputer

pipe_with_imputer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('model', LogisticRegression(max_iter=5000))
])

# Introduce missing values
X_train_missing = X_train.copy()
mask = np.random.random(X_train_missing.shape) < 0.05
X_train_missing[mask] = np.nan

pipe_with_imputer.fit(X_train_missing, y_train)

Exercise 30.14 — Examining grid search results

After running a GridSearchCV, examine grid_search.cv_results_ to create a heatmap of F1 scores across max_depth and n_estimators. Which hyperparameter has a larger effect on performance? Use pd.DataFrame(grid_search.cv_results_) to explore the results.

results_df = pd.DataFrame(grid_search.cv_results_)
# Look at columns: 'param_model__max_depth', 'param_model__n_estimators', 'mean_test_score'
# Pivot and create a heatmap with seaborn

Part C: Real-World Applications ⭐⭐⭐

Exercise 30.15 — Elena's complete pipeline

Elena has a dataset with numeric features (GDP, health spending, physicians, literacy rate, urban population) and categorical features (region: 'Africa', 'Americas', 'Europe', 'Asia', 'Oceania' and income_group: 'Low', 'Lower-Middle', 'Upper-Middle', 'High'). Design and build a complete pipeline that: 1. Handles both feature types with ColumnTransformer 2. Trains a random forest 3. Tunes hyperparameters with GridSearchCV 4. Evaluates on a held-out test set

Write the full code, including data splitting, pipeline definition, grid search, and evaluation.

Exercise 30.16 — Marcus's sales prediction pipeline

Marcus wants to predict daily sales using features: day_of_week (categorical), temperature (numeric), is_holiday (binary/numeric), month (categorical), and nearby_event (binary/numeric). Design the pipeline structure (you don't need actual data — describe each step in the pipeline and why it's needed).

Exercise 30.17 — Workflow critique

A colleague shows you this code and asks if it's correct:

# Scale features
X_scaled = StandardScaler().fit_transform(X)

# Feature selection
from sklearn.feature_selection import SelectKBest
selector = SelectKBest(k=5)
X_selected = selector.fit_transform(X_scaled, y)

# Split
X_train, X_test, y_train, y_test = train_test_split(X_selected, y)

# Train and evaluate
model = RandomForestClassifier()
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

Identify all the problems with this workflow and rewrite it using a proper pipeline.

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('selector', SelectKBest(k=5)),
    ('model', RandomForestClassifier(random_state=42))
])
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
pipe.fit(X_train, y_train)

Exercise 30.18 — Deployment scenario

You've built a pipeline that predicts customer churn and saved it with joblib. A software engineer asks: "What do I need to know to use this model in our web application?" Write a brief deployment guide covering: (1) what input the model expects, (2) what output it produces, (3) what happens if a new category appears that wasn't in the training data, and (4) when the model should be retrained.

Exercise 30.19 — Cost-sensitive grid search

For a medical screening problem where false negatives are 10x more costly than false positives, design a GridSearchCV that optimizes for recall rather than F1. Run it on the Breast Cancer dataset and compare the selected model to one optimized for F1. Do they choose different hyperparameters?

gs_recall = GridSearchCV(pipe, param_grid, cv=5, scoring='recall', n_jobs=-1)
gs_f1 = GridSearchCV(pipe, param_grid, cv=5, scoring='f1', n_jobs=-1)

Part D: Synthesis and Critical Thinking ⭐⭐⭐

Exercise 30.20 — End-to-end workflow design

You're starting a new project: predicting whether a restaurant will fail within its first two years. Your features include: location_type (urban/suburban/rural), cuisine_type (Italian/Mexican/American/etc.), owner_experience_years (numeric), initial_investment (numeric), nearby_competitors (numeric), avg_rent_sqft (numeric), population_density (numeric), and yelp_rating_month1 (numeric).

Design the complete workflow: data splitting strategy, preprocessing pipeline, models to try, hyperparameters to tune, evaluation metric(s), and how you'd report results. Justify each decision.

Exercise 30.21 — Pipeline vs. manual workflow debate

A colleague argues: "Pipelines are overengineered. I've been doing ML for years without them, and my models work fine." Write a thoughtful response (3-4 paragraphs) that acknowledges their experience while explaining the specific risks of working without pipelines. Include at least one example where the difference between a pipeline and manual workflow would lead to materially different results.

Exercise 30.22 — Feature engineering within pipelines

Discuss the challenges of doing feature engineering inside a pipeline. Specifically: (1) Can you do all feature engineering inside a pipeline? (2) What types of feature engineering might need to happen outside the pipeline? (3) How do you decide what goes inside vs. outside? Give examples.

Exercise 30.23 — Reflecting on Part V

Write a 1-page reflection on your journey through Part V (Chapters 25-30). Address: 1. What was the most surprising thing you learned? 2. What concept was hardest to understand, and how did you eventually get it? 3. How has your understanding of "what makes a good model" changed since Chapter 25? 4. If you had to explain the ML workflow to a friend in 5 sentences, what would you say?

Part E: Extension Challenges ⭐⭐⭐⭐

Exercise 30.24 — Nested cross-validation

Research nested cross-validation: an outer loop for estimating model performance and an inner loop for hyperparameter tuning. Implement it using scikit-learn on the Breast Cancer dataset. Compare the nested CV score to the non-nested (standard GridSearchCV) score. Why is the nested score usually lower? Write a paragraph explaining when nested CV is necessary.

from sklearn.model_selection import cross_val_score, GridSearchCV

inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

gs = GridSearchCV(pipe, param_grid, cv=inner_cv, scoring='f1', n_jobs=-1)
nested_scores = cross_val_score(gs, X, y, cv=outer_cv, scoring='f1')
print(f"Nested CV F1: {nested_scores.mean():.3f} +/- {nested_scores.std():.3f}")

Exercise 30.25 — Building a reusable ML template

Create a Python function or class called ModelComparer that: 1. Takes a list of (name, pipeline, param_grid) tuples 2. Runs GridSearchCV for each 3. Returns a summary DataFrame with best scores and parameters 4. Optionally evaluates the overall winner on a test set

Test it on the Iris dataset with at least 3 different model types.

class ModelComparer:
    def __init__(self, models, cv=5, scoring='f1'):
        self.models = models  # List of (name, pipeline, params) tuples
        self.cv = cv
        self.scoring = scoring
        self.results = {}

    def fit(self, X_train, y_train):
        for name, pipe, params in self.models:
            gs = GridSearchCV(pipe, params, cv=self.cv,
                              scoring=self.scoring, n_jobs=-1)
            gs.fit(X_train, y_train)
            self.results[name] = gs
        return self

    def summary(self):
        rows = []
        for name, gs in self.results.items():
            rows.append({
                'Model': name,
                'Best Score': gs.best_score_,
                'Best Params': gs.best_params_
            })
        return pd.DataFrame(rows).sort_values('Best Score', ascending=False)