Chapter 26: Key Takeaways — Explainable AI (XAI) and Model Governance

1. The Accountability Gap Cannot Be Closed by Performance Alone

A model's predictive accuracy — even an excellent 94% AUC — is necessary but not sufficient for regulatory approval in consequential decision contexts. When a model makes a decision that affects a customer, a financial position, or a regulatory obligation, someone in the organization must be able to explain why. This accountability cannot be transferred to the model itself. If the organization cannot explain what the model is doing, it cannot stand behind the decisions the model produces — and regulators will not accept "the algorithm decided" as a sufficient account of consequential automated decisions.

2. Multiple Regulatory Frameworks Demand Explainability

The explainability obligation is not confined to a single jurisdiction or a single law. In the US, ECOA/Regulation B requires specific adverse action reasons for declined credit applications; Federal Reserve SR 11-7 requires conceptual soundness, ongoing monitoring, and outcomes analysis for all models. In the EU, GDPR Article 22 creates rights against solely automated decisions with significant effects and a right to a meaningful explanation; the EU AI Act classifies credit scoring, fraud detection, and AML systems as high-risk AI with transparency, documentation, and human oversight requirements. In the UK, the FCA's Consumer Duty framework requires demonstrably fair outcomes across consumer groups. These frameworks overlap and reinforce each other.

3. SR 11-7's Three Pillars Remain the Foundational U.S. Framework

The Federal Reserve's Supervisory Guidance on Model Risk Management, SR 11-7, rests on three pillars that continue to define U.S. expectations for model governance at large financial institutions: conceptual soundness (the model's theoretical basis must be sound and appropriate for its purpose), ongoing monitoring (performance, population stability, and data quality must be tracked continuously in production), and outcomes analysis (model predictions must be compared to actual outcomes over time). These three pillars apply regardless of model type — they predate machine learning but apply fully to gradient-boosted trees, neural networks, and ensemble methods.

4. A Model Inventory Is the Non-Negotiable Starting Point

Every model in production must be registered in a centralized inventory before it can be governed. The inventory must capture model purpose, owner, type, training data, validation status, performance metrics, PSI, and review dates. The most common governance failure is not poor validation of registered models — it is models that were never registered at all. Shadow models, Excel-based analytical tools, and vendor-supplied scoring systems that the firm's compliance function never knew were deployed are the most significant blind spots. An effective model governance program begins with a comprehensive model discovery exercise, not with a governance policy for models already known to exist.

5. The Population Stability Index Is the Primary Early Warning for Model Decay

The Population Stability Index (PSI) measures whether the distribution of model inputs or outputs in the current production population has shifted from the training population. The standard thresholds are: PSI below 0.10 is stable and requires no immediate action; PSI between 0.10 and 0.25 indicates a minor shift that warrants increased monitoring frequency and investigation; PSI above 0.25 is a critical breach that should trigger suspension of the model from high-stakes decisions, escalation to model risk management, and initiation of a retraining investigation. PSI monitoring must begin from the day a model enters production, not after a performance problem has become apparent.

6. SHAP Is the Gold Standard for Regulatory Explainability of Tabular Models

SHAP (SHapley Additive exPlanations), grounded in cooperative game theory, computes the exact marginal contribution of each feature to an individual prediction. For tree-based models (XGBoost, LightGBM, gradient boosted trees), TreeSHAP computes these values in polynomial time, making real-time explanation computationally feasible. SHAP satisfies formal theoretical axioms — local accuracy, missingness, consistency — that make its attributions principled rather than approximate. SHAP values can be used directly to generate Regulation B adverse action reasons (the features with the largest negative SHAP values are the primary drivers of decline) and to satisfy GDPR Article 22 explanation requirements. For regulatory documentation of individual credit decisions, SHAP is the appropriate technical standard.

7. LIME Is Useful for Exploration but Insufficient for Regulatory Documentation

LIME (Local Interpretable Model-agnostic Explanations) trains a simple linear approximation in the local neighborhood of each prediction and treats the linear model's coefficients as explanations. LIME is model-agnostic and faster than kernel SHAP for non-tree models, making it useful for exploratory analysis and rapid prototyping. However, LIME explanations are unstable — different runs on the same instance with different random seeds can produce meaningfully different results. This instability makes LIME unsuitable for regulatory documentation of individual adverse action decisions, where the firm may be required to defend the specific reasons provided to a customer. Use LIME for exploration; use SHAP for regulatory deliverables.

8. Adverse Action Reasons Must Reflect the Model's Actual Decision Drivers

Regulation B's adverse action notice requirement is not satisfied by generic or vague statements. The reasons provided to an applicant who was declined must reflect the actual factors that drove the model's output for that specific application — not the generally important features of the model across the population, and not standard categories that happen to apply to the applicant. SHAP-based adverse action coding, which maps the top negative SHAP values for a given instance to plain-language reason statements, is the technically sound method for meeting this requirement. The mapping between SHAP values and reason statements must itself be documented and validated, because a technically correct SHAP value translated into a misleading reason statement does not satisfy the regulation.

9. Models Cannot Optimize All Fairness Metrics Simultaneously

A mathematical result established in the fairness literature proves that demographic parity (equal approval rates across groups), equalized odds (equal true positive rates and false positive rates across groups), and counterfactual fairness cannot all be satisfied simultaneously when base rates differ across groups — which they almost always do in practice. Model governance teams must therefore make explicit, documented choices about which fairness objectives are prioritized for a given model, and why. The choice is not merely technical; it reflects value judgments about which types of fairness errors are more harmful. These choices must be disclosed to and accepted by the model risk committee and, where relevant, defended to regulators.

10. The EU AI Act Introduces CE Marking and Conformity Assessment for High-Risk AI

Under the EU AI Act, financial services firms deploying high-risk AI systems — which include credit scoring, fraud detection, AML scoring, and insurance pricing — must not only meet the Act's substantive requirements (risk management system, data governance, technical documentation, record-keeping, transparency, human oversight, accuracy and robustness) but must in many cases submit to a conformity assessment involving a notified body before the system is placed on the EU market. This requirement makes model governance a pre-market obligation, not a post-deployment compliance exercise. Firms with EU operations should treat the EU AI Act's Annex III categorization as a design constraint from the earliest stage of model development, not as a box to check after the model has been built.