Chapter 26: Exercises — Explainable AI (XAI) and Model Governance

Exercise 1: Interpreting a SHAP Waterfall Chart

The following table describes the SHAP waterfall chart for a consumer credit decision — application ID APP-20240302-1147. The model is a gradient-boosted tree classifier. The base value (model output for the average instance) is 0.44. The final predicted score is 0.29. The decision threshold is 0.50. The application was declined.

Task 1.1 — Mathematical check: Confirm that the SHAP values and base value are internally consistent with the final predicted score. Show your calculation. (Hint: SHAP values + base value = predicted score for a well-formed SHAP explanation.)

Task 1.2 — Adverse action reasons: Based on the SHAP values, identify the three features that most strongly pushed this application toward decline and formulate plain-language adverse action reasons for each, suitable for inclusion in a Regulation B adverse action notice.

Task 1.3 — Counterfactual reasoning: The applicant calls Cornerstone's customer service line to ask what they could do differently to receive approval. Using the waterfall chart, describe two plausible counterfactual paths to approval — one that primarily addresses the payment history, one that primarily addresses the debt-to-income ratio — and what each would require.

Task 1.4 — Model governance question: A compliance officer asks: "If the model changed — if we retrained it next month — would the SHAP values for this exact application be different?" How would you answer this question, and what are the implications for adverse action records that firms maintain?

Exercise 2: Designing a Model Inventory

Rivergate Financial Services is a mid-sized UK challenger bank with the following quantitative systems currently in use. Your task is to design a complete model inventory for Rivergate.

Systems identified in discovery exercise:

1. A logistic regression model scoring credit applications for personal loans, predicting 12-month probability of default. Built in-house. In production since March 2022. Used to approve or decline 100% of applications without human review.

2. A vendor-supplied anti-money laundering transaction monitoring system. Vendor: Actimize. Rivergate's compliance team receives alerts and investigates; the vendor system scores each transaction. Rivergate has limited visibility into the methodology.

3. An Excel spreadsheet maintained by the treasury team that computes liquidity coverage ratio and reports a daily LCR figure used in the firm's regulatory submission. Uses regulatory formula exactly.

4. A Python script written by a data analyst that generates a weekly customer segmentation based on product holdings, recency of transactions, and balance levels. The segments are used by the marketing team to target communications. No adverse decisions are based on it.

5. A gradient-boosted tree model that scores savings account applications for fraud risk. Built by the fraud team. In production since January 2024. High-risk applications are declined automatically; medium-risk applications are referred to a human analyst.

6. A Monte Carlo simulation model used by the market risk team to compute Value at Risk for the trading book. Outputs feed directly into capital calculations and regulatory reporting.

7. A rules-based decision engine for customer onboarding that checks whether applicants appear on sanctions lists and PEP databases. Implemented as a vendor API.

Task 2.1 — Inventory classification: For each of the seven systems above, determine: (a) whether it qualifies as a "model" under SR 11-7's definition, with justification; (b) if so, the appropriate tier (Tier 1, 2, or 3); and (c) the primary regulatory framework most relevant to its governance.

Task 2.2 — Inventory record design: Design the fields for a model inventory record that would capture all information needed to govern these models. For each field, specify: field name, data type, and whether it is mandatory or optional. Your record should have at least 15 fields.

Task 2.3 — Priority order: Rank the seven systems in order of the governance urgency — which should be validated, documented, and formally registered first? Justify your ranking with reference to risk level and regulatory consequences of governance failure.

Exercise 3: Calculating Population Stability Index

A consumer credit scorecard was trained on application data from 2020 to 2022. The validation team is conducting its annual PSI review using the past three months of production scores.

The training data produced the following score distribution (100,000 applicants):

The current quarter's production population produced the following score distribution (18,000 applicants):

The PSI formula is:

PSI = sum over all bands of: (Current% - Training%) × ln(Current% / Training%)

Task 3.1: Calculate the PSI value for this model. Show your work for at least four score bands before presenting the total.

Task 3.2: Interpret the PSI result using standard thresholds (below 0.10 = stable; 0.10–0.25 = minor shift; above 0.25 = critical). What action, if any, is required?

Task 3.3: Looking at the score band data, what has happened to the score distribution? Is the population scoring higher or lower on average than the training population? What could cause this kind of shift in a consumer credit context?

Task 3.4: If you were the model risk manager receiving this PSI report, write a 150-word escalation memo to the model's business owner describing the finding and the required response.

Exercise 4: Implementing Disparate Impact Analysis

Implement the following Python function that computes a disparate impact report for a credit decision model.

from __future__ import annotations
import pandas as pd
import numpy as np
from dataclasses import dataclass


@dataclass
class DisparateImpactResult:
    """Result of a disparate impact analysis."""
    group_column: str
    outcome_column: str
    group_stats: pd.DataFrame      # One row per group: approval_rate, n_applications, n_approved
    disparate_impact_ratios: dict[str, float]  # group -> DI ratio vs. reference group
    reference_group: str           # Group with highest approval rate (or specified group)
    four_fifths_violations: list[str]  # Groups with DI ratio < 0.80

    def summary(self) -> str:
        """Return a human-readable summary of the disparate impact analysis."""
        # TODO: implement in the exercise
        raise NotImplementedError


def calculate_disparate_impact_ratio(
    df: pd.DataFrame,
    group_column: str,
    outcome_column: str,
    reference_group: str | None = None,
    threshold: float = 0.80,
) -> DisparateImpactResult:
    """
    Calculate disparate impact ratios for all groups in a categorical column.

    Args:
        df:               DataFrame containing application-level data.
        group_column:     Column name identifying the group for analysis
                          (e.g., "postcode_area", "age_band", "employment_type").
        outcome_column:   Column name for the binary outcome (1 = approved, 0 = declined).
        reference_group:  The group to use as the reference (denominator) for ratios.
                          If None, uses the group with the highest approval rate.
        threshold:        The four-fifths rule threshold (default 0.80).

    Returns:
        DisparateImpactResult with approval rates and disparate impact ratios.

    Example:
        >>> data = pd.DataFrame({
        ...     "postcode_area": ["N1", "N1", "SE1", "SE1", "SE1", "E1"],
        ...     "approved":      [1,    0,    1,     1,     0,    0  ],
        ... })
        >>> result = calculate_disparate_impact_ratio(data, "postcode_area", "approved")
        >>> print(result.disparate_impact_ratios)
        # {'SE1': 1.0, 'N1': 0.667, 'E1': 0.0}  (reference = SE1 with 67% approval)
    """
    # YOUR IMPLEMENTATION HERE
    pass


def implement_summary_method(result: DisparateImpactResult) -> str:
    """
    Implement the summary() method for DisparateImpactResult.
    The summary should return a formatted string that includes:
    - The analysis parameters (which column, which outcome)
    - A table of all groups with their approval rates and DI ratios
    - Identification of any four-fifths rule violations
    - A plain-English interpretation

    YOUR IMPLEMENTATION HERE
    """
    pass

Task 4.1: Implement the calculate_disparate_impact_ratio function. Your implementation should: - Compute the approval rate for each group in group_column. - Determine the reference group (highest approval rate if not specified). - Compute the disparate impact ratio for each group as: group_approval_rate / reference_approval_rate. - Identify which groups have a DI ratio below the threshold (four-fifths violations). - Return a fully populated DisparateImpactResult.

Task 4.2: Implement the summary() method on DisparateImpactResult.

Task 4.3: Test your implementation with the following synthetic dataset. Describe what you find.

# Test dataset
np.random.seed(42)
n = 5000

test_data = pd.DataFrame({
    "region": np.random.choice(
        ["London-North", "London-East", "London-South", "Midlands", "North"],
        size=n,
        p=[0.30, 0.20, 0.15, 0.20, 0.15]
    ),
    "approved": np.where(
        np.random.choice(["London-North", "London-East", "London-South", "Midlands", "North"],
                         size=n, p=[0.30, 0.20, 0.15, 0.20, 0.15]) == "London-North",
        np.random.binomial(1, 0.78, n),
        np.where(
            np.random.choice(["London-North", "London-East", "London-South", "Midlands", "North"],
                             size=n, p=[0.30, 0.20, 0.15, 0.20, 0.15]) == "North",
            np.random.binomial(1, 0.55, n),
            np.random.binomial(1, 0.70, n)
        )
    )
})

result = calculate_disparate_impact_ratio(test_data, "region", "approved")
print(result.summary())

Task 4.4 — Regulatory interpretation: If the four-fifths rule is violated for one region in your test results, what are the next steps a financial institution should take? Is a four-fifths violation proof of illegal discrimination? Explain.

Exercise 5: Drafting Adverse Action Reasons

For each of the following three credit decline scenarios, draft the adverse action reasons that would appear on a Regulation B adverse action notice. Your reasons must be specific, accurate to the SHAP values provided, and written in plain English suitable for a customer to understand. Each scenario should produce three reasons.

Scenario A — Thin File Applicant

Applicant: 23-year-old, first-time borrower, requesting a £5,000 personal loan. Model score: 0.27 (threshold: 0.50). Decision: Declined.

SHAP values (most negative to least negative): - credit_history_months (8 months): -0.11 - num_existing_accounts (0): -0.09 - income (£19,500): -0.05 - account_age_months (8): -0.04 - debt_to_income (0.38): -0.03 - employment_months (14): -0.02 - credit_score (551): -0.02

Scenario B — Over-Extended Borrower

Applicant: 41-year-old, existing customer, requesting a £25,000 home improvement loan. Model score: 0.33 (threshold: 0.50). Decision: Declined.

SHAP values (most negative to least negative): - debt_to_income (0.68): -0.13 - missed_payments_12m (1): -0.07 - outstanding_loan_balance (£42,000): -0.06 - num_recent_inquiries (4): -0.05 - loan_to_income_ratio (6.4): -0.04 - income (£39,000): -0.03 - credit_score (631): +0.02 - existing_customer (1): +0.03

Scenario C — Recent Credit Events

Applicant: 35-year-old, requesting a £10,000 personal loan. Previously a good credit customer. Model score: 0.21 (threshold: 0.50). Decision: Declined.

SHAP values (most negative to least negative): - missed_payments_24m (4): -0.15 - county_court_judgments (1): -0.12 - missed_payments_12m (2): -0.09 - credit_score (543): -0.07 - debt_to_income (0.51): -0.04 - account_age_months (72): +0.05 - existing_customer (1): +0.04 - income (£34,000): +0.02

Guidance: Adverse action reasons under Regulation B must (1) be specific, not vague; (2) accurately reflect actual decision drivers; (3) be written in plain English comprehensible to an applicant with no financial background; and (4) not include legally protected characteristics or their proxies. For each scenario, also note whether there are any features in the SHAP values that might raise fairness or proxy concerns if included in an adverse action notice.