Exercises: Chapter 19

Model Interpretation

Exercise 1: SHAP Fundamentals (Conceptual)

a) In your own words, explain the difference between interpretability and explainability. Give one example of a model that is inherently interpretable and one that requires post-hoc explainability methods.

b) A Shapley value for feature $j$ represents the "fair contribution" of that feature to a prediction. What does "fair" mean in this context? What property of Shapley values guarantees that the contributions sum to the total prediction minus the baseline?

c) Explain the difference between global and local interpretation. For each of the following audiences, state whether they primarily need global interpretation, local interpretation, or both, and explain why: - A VP of Product evaluating whether to deploy the model - A customer success representative using the model daily - A data scientist validating the model before deployment - A regulator auditing the model for compliance

d) Why is model.feature_importances_ (gain-based importance) insufficient for most interpretation tasks? Name three specific limitations that SHAP addresses.

Exercise 2: Computing and Interpreting SHAP Values (Code)

Using the StreamFlow churn dataset and model from the chapter, complete the following:

a) Compute SHAP values for the test set using TreeExplainer. Verify the additivity property: for any observation, confirm that the sum of SHAP values plus the expected value equals the model's raw prediction (in log-odds).

import numpy as np
import pandas as pd
import shap
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split

# --- Set up the StreamFlow dataset (use the chapter's setup code) ---
# Your code here: load data, train model, compute SHAP values

# Verify additivity for observation 0
idx = 0
raw_pred = model.predict(X_test.iloc[[idx]], output_margin=True)[0]
shap_sum = shap_values[idx].sum() + explainer.expected_value
print(f"Raw prediction: {raw_pred:.6f}")
print(f"SHAP sum:       {shap_sum:.6f}")
print(f"Match: {np.isclose(raw_pred, shap_sum, atol=1e-4)}")

b) Create a SHAP summary plot (dot plot) for the test set. Identify the top 5 features and, for each, describe the direction of its effect on churn probability based on the color pattern.

c) Create waterfall plots for three observations: - One with a very high predicted churn probability (>60%) - One near the decision boundary (45-55%) - One with a very low predicted churn probability (<15%)

For each, list the top 3 features driving the prediction and explain in plain English why the model made that prediction.

Exercise 3: SHAP Dependence Plots and Interactions (Code + Analysis)

a) Create a SHAP dependence plot for days_since_last_session with the automatic interaction feature. Describe what the plot reveals about: - The shape of the relationship (linear, threshold, etc.) - Any saturation effects (does the effect plateau?) - The interaction feature selected by SHAP

b) Create SHAP dependence plots for payment_failures_6m and monthly_hours_watched. Compare the three relationships. Which feature has the most linear relationship with its SHAP values? Which has the most non-linear relationship?

c) Create a 2-feature dependence analysis by coloring the days_since_last_session dependence plot by payment_failures_6m explicitly:

shap.dependence_plot(
    "days_since_last_session", shap_values, X_test,
    interaction_index="payment_failures_6m", show=False
)

Describe the interaction. Does having more payment failures amplify or dampen the effect of inactivity?

Exercise 4: Partial Dependence and ICE Plots (Code + Analysis)

a) Create partial dependence plots for the top 6 features from the SHAP analysis. For each, describe: - The overall shape (monotonic increasing, decreasing, U-shaped, etc.) - Any threshold effects (abrupt changes) - The range of the partial dependence (how much does the prediction change?)

b) For days_since_last_session, create a combined PDP + ICE plot using kind='both' with 200 sampled observations. Answer: - Are the ICE lines roughly parallel? What does this mean about the consistency of the feature's effect? - Are there any ICE lines that deviate strongly from the average PDP? What might explain those cases?

c) Create a 2D partial dependence plot for the pair (days_since_last_session, monthly_hours_watched). Describe the interaction surface. Is there a region where churn risk is particularly high? Particularly low?

d) Explain why the PDP for a feature might be flat even though the feature has high permutation importance. Construct a hypothetical scenario (you do not need code) where this occurs.

Exercise 5: Permutation Importance (Code + Analysis)

a) Compute permutation importance for the StreamFlow model on the test set with n_repeats=30. Create a bar chart of the top 10 features.

b) Compare the permutation importance ranking to the SHAP-based ranking (mean |SHAP|) and the built-in feature importance (gain). Create a table showing all three rankings side by side.

comparison = pd.DataFrame({
    'feature': X_test.columns,
    'shap_rank': # your code,
    'perm_rank': # your code,
    'gain_rank': # your code,
})

c) For any features where the rankings disagree substantially (differ by more than 3 positions), explain what might cause the disagreement. Consider: - Correlated features - Non-linear relationships - Interaction effects

d) Demonstrate the correlated-features problem with permutation importance: - Add a column sessions_copy that is a noisy copy of sessions_last_30d (add small random noise). - Retrain the model with the extra feature. - Compute permutation importance. - Show that both sessions_last_30d and sessions_copy have lower permutation importance than the original sessions_last_30d had without the copy.

Exercise 6: LIME (Code + Comparison)

a) Install and use LIME to explain the same three observations you explained with SHAP in Exercise 2c. For each observation, print the top 5 features from LIME.

b) Compare the LIME explanations to the SHAP explanations. For each observation, answer: - Do the top 3 features agree between LIME and SHAP? - Do the directions (increases/decreases churn) agree? - Where do they disagree, and what might explain the disagreement?

c) Demonstrate LIME's instability: run LIME on the same observation 5 times (without setting random_state) and record the top 3 features each time. Do the rankings change? Do the weights change?

d) Now set random_state=42 and num_samples=10000. Run LIME 5 times. Is it now stable? What is the tradeoff of using a large num_samples?

Exercise 7: Translating for Stakeholders (Written)

a) A product manager asks: "The model has an AUC of 0.94. What does that mean in plain English?" Write a 2-sentence response that does not use the term "AUC," "ROC," or any statistical jargon.

b) Write a 3-bullet executive summary of the StreamFlow churn model's interpretation results, suitable for a VP-level audience. Each bullet should be one sentence, use no technical jargon, and be actionable.

c) A customer success representative calls you and says: "The model flagged Customer #12847 as high-risk, but I talked to them last week and they seemed happy. Is the model wrong?" Write a 4-sentence response that acknowledges their concern, explains what the model can and cannot see, and suggests a next step.

d) Design a one-page "How to Read This Report" guide for a hospital readmission model (not the churn model). The audience is registered nurses who review discharge plans. The guide should explain: - What the risk score means - How to read the "top 3 reasons" for a flagged patient - What to do when the model's reasons do not match their clinical judgment - What the model cannot see

Exercise 8: Comprehensive Interpretation Report (Project)

Build a complete interpretation report for a model of your choice. You may use the StreamFlow churn model or one of your own models. The report must include:

a) Global interpretation: - SHAP summary plot (dot version) - Permutation importance bar chart - A written comparison of the two methods' rankings

b) Feature-level analysis: - PDP + ICE plots for the top 3 features - SHAP dependence plots for the top 3 features - Written description of the relationships (linear? threshold? saturating?)

c) Local explanations: - SHAP waterfall plots for 3 representative observations (high-risk, medium-risk, low-risk) - A plain-English table translating each waterfall into "top 3 reasons"

d) Stakeholder deliverable: - A one-page "How to Read This" guide written for a non-technical user of the model

e) Reflection: - Did any interpretation result surprise you? - Did the interpretation reveal any potential problems with the model (e.g., a feature that should not be used, an unexpected interaction)? - What would you change about the model based on what you learned?

Exercise 9: Edge Cases and Limitations (Short Answer)

a) A colleague computes SHAP values for a linear regression model and says: "The SHAP values are just the coefficients times the feature values." Is this correct? Explain why or why not.

b) You compute a PDP for income and it shows that higher income increases the probability of loan default. This contradicts domain knowledge. Name three possible explanations that do not involve the PDP being "wrong."

c) A team member says: "SHAP tells us which features are important, so we can remove the unimportant features and retrain." Is this good advice? Under what circumstances might removing low-SHAP features hurt performance?

d) You are explaining a model to a regulator who asks: "Can you guarantee that the model does not use age in its predictions?" You did not include age as a feature, but years_since_first_purchase is in the model and correlates at r=0.89 with age. How do you respond?

Exercise 10: SHAP for Multi-Class Classification (Challenge)

Extend the interpretation workflow to a multi-class problem:

a) Create a synthetic dataset with 3 classes. Train a multi-class XGBoost model.

b) Compute SHAP values. Note that for multi-class problems, shap_values is a list of arrays (one per class). Explain what the SHAP values for class $k$ represent.

c) Create summary plots for each class. Do the same features dominate for all classes, or do different classes have different important features?

d) Create a waterfall plot for a single observation, showing the SHAP contributions for the predicted class. Write a plain-English explanation of why the model chose that class.

These exercises support Chapter 19: Model Interpretation. Return to the chapter for full context.