Case Study 14.1: SHAP in Practice — How a Bank Used Explainability to Detect Hidden Discrimination

Type: Composite illustrative scenario based on documented industry practices, regulatory guidance (SR 11-7, ECOA/Regulation B, OCC guidance on model risk management), and published research on proxy discrimination in credit models. All institution names, individuals, and specific figures are illustrative.

Background

Meridian Community Bank, a mid-sized regional institution with approximately $18 billion in assets and a mortgage loan portfolio spanning three metropolitan statistical areas in the Mid-Atlantic United States, had operated a traditional logistic regression-based mortgage underwriting model for eleven years. The model used a relatively small feature set: FICO score, debt-to-income ratio, loan-to-value ratio, employment tenure, and loan type. Loan officers understood the model's behavior, compliance staff could generate clear adverse action notices, and the model had survived several regulatory examinations without material findings.

In 2019, Meridian's chief risk officer commissioned a model modernization initiative. A newly hired data science team benchmarked the legacy model against several machine learning alternatives and found that a gradient-boosted tree ensemble (XGBoost) trained on an expanded feature set reduced the model's expected loss rate by approximately 4.3 percentage points — a difference that, across Meridian's loan volume, was projected to be worth tens of millions of dollars annually in avoided defaults. The expanded feature set included the traditional variables plus: credit utilization history over rolling 12-month windows, trade line age distribution, inquiry counts, payment velocity metrics, and — critically — an aggregated neighborhood risk score derived from the applicant's five-digit zip code.

After internal validation by the model risk management (MRM) team, the new model was approved for deployment in September 2020. Adverse action notices for denied applications were generated using a LIME-based wrapper that identified the top four adverse factors for each denial. Loan volumes processed by the new model grew steadily, and the model's performance metrics remained within expected ranges through 2021.

The Regulatory Examination

In late 2021, the Office of the Comptroller of the Currency (OCC) conducted a full-scope examination of Meridian's model risk management practices, with particular attention to the 2020 model modernization. The examination team, drawing on SR 11-7 — the Federal Reserve and OCC's joint guidance on model risk management — raised several concerns.

SR 11-7 requires that institutions maintain effective governance of model development, validation, and ongoing monitoring. For models used in credit decisions, this includes analysis of whether model outputs produce disparate impact on protected classes under the Equal Credit Opportunity Act and the Fair Housing Act. The examination team noted that Meridian's validation documentation addressed traditional performance metrics (AUC-ROC, KS statistic, Gini coefficient, stability index) but did not include a systematic analysis of whether the zip-code-based neighborhood risk score was functioning as a proxy for race or national origin.

The examiners also raised a concern about the LIME-based adverse action notice methodology. SR 11-7 requires that model risk management include validation of model outputs, not just model performance — and for explanation tools used in compliance processes, validation means verifying that the explanations faithfully represent the model's actual decision logic. Meridian's model validation documentation did not include this type of faithfulness testing for the LIME wrapper.

The examination concluded with a Matters Requiring Attention (MRA) — a formal supervisory finding requiring remediation — directing Meridian to: (1) conduct a comprehensive disparate impact analysis of the new model, with particular attention to the zip-code-derived feature; (2) validate the faithfulness of the LIME-based adverse action methodology; and (3) document findings and remediation plans within 180 days.

Deploying SHAP for the Audit

Meridian's model risk management team, working with its data science group and external model risk consultants, chose SHAP as the primary analytical tool for the mandated audit. The choice reflected several considerations: SHAP's stronger mathematical foundations relative to LIME, its ability to support both global and local analysis, and its suitability for detecting proxy variable usage through aggregate feature attribution patterns.

The team computed TreeSHAP values for the full population of loan applications processed by the new model from October 2020 through December 2021 — approximately 14,800 complete applications, of which 3,240 resulted in denials. For each application, SHAP values were computed for each of the model's 23 input features, providing a complete attribution of every denial to its contributing factors.

Global Analysis: What the Beeswarm Plot Revealed

The global SHAP beeswarm plot immediately surfaced a pattern that the standard model validation had not caught. The neighborhood risk score — the zip-code-derived feature — appeared as the second most influential feature in the model by mean absolute SHAP value, behind only credit utilization history. More notably, the beeswarm plot revealed that the feature's SHAP contributions were almost entirely positive for applications from zip codes with scores above the 60th percentile: applicants from those zip codes were almost uniformly pushed toward denial by this feature, with SHAP contributions as high as +0.19 in log-odds units.

This pattern would not have been apparent from a simple adverse action notice analysis, which only showed the top four factors for individual denials. The aggregate view made visible what the individual view obscured.

Proxy Analysis: Connecting Zip Code to Race

The audit team obtained census block group demographic data from the 2020 American Community Survey (ACS) and matched it to the zip codes in the loan application dataset. The analysis examined the correlation between the neighborhood risk score's SHAP contribution and the percentage of non-Hispanic Black and Hispanic residents in the applicant's census tract.

The findings were uncomfortable. The Pearson correlation between an application's neighborhood risk score SHAP value and the non-Hispanic Black + Hispanic percentage of its associated census tract was r = 0.47 — a moderate-to-strong positive correlation indicating that applicants from majority-minority neighborhoods were receiving substantially higher SHAP contributions (pushing toward denial) from the zip-code feature than applicants from predominantly white neighborhoods. Controlling for individual creditworthiness factors (FICO score, DTI, LTV), applicants from census tracts with greater than 60% non-Hispanic Black or Hispanic population were approximately 1.8 times more likely to receive a neighborhood risk score SHAP value above the 75th percentile.

The Redlining Connection

The audit team's follow-on analysis examined the construction of the neighborhood risk score itself. The score had been purchased from a third-party data vendor as an aggregated measure of neighborhood financial health, incorporating variables such as local foreclosure rates, median credit scores in the area, small business loan default rates, and housing price volatility indices.

The historical structure of these variables is not accidental. Decades of federal redlining policy — the explicit exclusion of minority neighborhoods from federally backed mortgage insurance from the 1930s through the 1960s, followed by decades of discriminatory private lending practices — systematically denied wealth-building investment to majority-minority neighborhoods. The result was a legacy of lower property values, higher foreclosure vulnerability, lower local credit scores, and higher financial volatility in precisely those neighborhoods that had been targeted for exclusion. A score trained to predict financial risk from these variables would, as a predictable consequence of this history, score majority-minority neighborhoods as higher-risk — not because their residents are less creditworthy, but because the neighborhoods themselves bear the financial scars of government-sanctioned discrimination.

The neighborhood risk score was, in the audit team's assessment, a sophisticated form of redlining — not in intent, but in effect. The model had learned to use zip code as a proxy for race and was systematically disadvantaging applicants from communities that had already been disadvantaged by prior discriminatory policy. The fact that the model was never told about race was legally and ethically irrelevant: ECOA disparate impact analysis focuses on outcomes, not intent.

The Legal Exposure

Outside counsel was engaged to assess the legal implications of the audit findings. Their analysis identified exposure under two statutory frameworks.

ECOA Disparate Impact. Under the Supreme Court's interpretation of ECOA and the Fair Housing Act, facially neutral lending policies that have a statistically significant disparate impact on a protected class are unlawful unless the institution can demonstrate that the policy is necessary to meet a compelling business need that cannot be met by a less discriminatory alternative. The neighborhood risk score's observed impact on minority applicants — a denial rate roughly 1.8 times higher for applications from majority-minority neighborhoods, after controlling for individual creditworthiness factors — was likely sufficient to establish a prima facie disparate impact claim.

The business necessity analysis was uncertain. The neighborhood risk score did provide independent predictive value beyond individual credit variables, but the magnitude of that incremental value relative to the discriminatory effect was difficult to defend. Critically, the less discriminatory alternative prong was problematic: the audit team's analysis found several alternative aggregate features — local employment rate trends, distance to nearest bank branch, regional home price index — that provided comparable predictive lift without the same degree of demographic correlation.

CFPB Adverse Action Notice Concerns. The CFPB's examination of the LIME-based adverse action methodology also revealed deficiencies. The LIME wrapper's instability meant that different runs produced different adverse action reasons for the same application — a reproducibility problem that creates regulatory risk when applicants request explanation of their denial. Additionally, the audit found that neighborhood risk score appeared as an adverse factor in adverse action notices with insufficient frequency relative to its actual SHAP contribution, suggesting that the LIME approximation was not accurately reflecting the model's behavior for this feature.

Remediation

Meridian's remediation plan, submitted to the OCC within the 180-day window, addressed both the model and the explanation methodology.

Phase 1: Removing the Neighborhood Risk Score

The model was retrained without the neighborhood risk score and any other features that the audit identified as having high demographic correlation. The performance impact was modest: the new model's AUC-ROC was 0.003 lower than the model with the zip-code feature. Outside counsel and the model risk team jointly assessed this difference as insufficient to constitute a compelling business necessity justification for retaining the discriminatory feature.

Phase 2: Alternative Feature Development

The data science team developed several alternative aggregate features with lower demographic correlation, including: an applicant-level employment trajectory score (directional trend in employment history), a local housing market stability index derived from regional price indices with lower correlation to historical redlining geography, and an expanded credit utilization trend feature derived from individual credit bureau data rather than zip-code aggregates. These features provided comparable incremental lift to the neighborhood risk score with substantially lower demographic correlation, satisfying the less discriminatory alternative analysis.

Phase 3: Replacing LIME with SHAP for Adverse Action Notices

The LIME-based adverse action notice methodology was replaced with a SHAP-based system. Because SHAP values are guaranteed to sum to the difference between the prediction and the baseline, and because TreeSHAP provides exact (not approximate) values, the SHAP-based system was more reproducible and more faithful to the model's actual behavior. The top four adverse action reasons were derived from the SHAP waterfall values for each denied application, with a mapping layer that translated SHAP feature names into Regulation B-compliant factor descriptions (e.g., "credit_utilization_12m_max" was mapped to "Insufficient reduction in credit card balances over the past 12 months").

The validation team confirmed that the new adverse action notice methodology was more faithful by testing a sample of explanations against the sanity checks described in Adebayo et al. (2018): SHAP values changed substantially when the model was retrained on randomized labels, confirming that the explanations reflected model behavior rather than input data structure.

Phase 4: Post-Remediation SHAP Analysis

A full SHAP audit of the remediated model was conducted on a holdout validation set. The beeswarm plot of the new model no longer showed the neighborhood risk score as a top feature (it had been removed). The correlation between demographic composition and the SHAP contributions of remaining features was substantially lower: r = 0.11 between census tract minority percentage and the top feature's SHAP contribution, compared to r = 0.47 in the original model.

The denial rate gap between majority-minority and majority-white neighborhoods narrowed from 1.8x to 1.2x after controlling for individual creditworthiness factors — still above 1.0 (reflecting genuine differences in financial profile distributions across neighborhoods, themselves a legacy of historical discrimination), but substantially reduced and below the level that external counsel assessed as likely to give rise to a successful disparate impact claim.

Regulatory Disclosure and Ongoing Monitoring

Meridian filed a voluntary disclosure with the CFPB and the OCC summarizing the audit findings, the identified disparate impact, and the remediation plan. The disclosure was well-received: regulators responded that Meridian's proactive identification of the problem, combined with a documented and validated remediation plan, placed the institution in a substantially different posture than institutions that only address bias findings under compulsion.

Ongoing monitoring requirements were incorporated into the model's governance documentation:

• Quarterly SHAP audit: A full SHAP analysis is run quarterly on the prior quarter's loan decisions, with automated alerts if any feature's mean absolute SHAP value exceeds defined thresholds or if demographic correlations with feature contributions exceed specified bounds.
• Annual full disparate impact analysis: A complete ECOA disparate impact analysis, using matched-pair analysis and regression-based methods, is conducted annually and presented to the model risk committee and the board's audit and compliance committee.
• Adverse action notice sampling: A monthly sample of adverse action notices is reviewed for faithfulness: a data scientist manually verifies that the SHAP-derived reasons for a sample of denials accurately describe the model's behavior for those applications.

What This Case Illustrates

Several lessons from Meridian's experience are worth drawing explicitly.

SHAP as a global audit tool. The proxy discrimination that SHAP identified was not visible in individual adverse action notices, model performance metrics, or standard model validation approaches. It became visible only when SHAP values were aggregated across the full application population and cross-referenced with demographic data. This is the most powerful application of SHAP in governance contexts: not explaining individual decisions, but revealing systemic patterns in model behavior.

The historical context of technical variables. The neighborhood risk score was a technically derived, commercially available data product. Its discriminatory impact did not arise from anyone's intent to discriminate; it arose from the fact that the variables it aggregated bear the imprint of historical discrimination. Technical neutrality is not a defense under disparate impact analysis, and organizations must evaluate their data inputs not just for predictive validity but for potential to encode and perpetuate historical injustice.

Explanation versus accountability. The LIME-based adverse action notices in Meridian's original model were a transparency mechanism. They did not prevent the discriminatory outcome. The SHAP audit did not produce accountability by itself either — that required the regulatory finding, the outside counsel engagement, the board oversight, and the documented remediation. Explanation tools create the information necessary for accountability; they do not create accountability automatically.

The accuracy-fairness tradeoff in practice. The remediated model was marginally less accurate (0.003 AUC-ROC lower) than the discriminatory model. This tradeoff — small accuracy cost, substantial reduction in discriminatory impact — is precisely what disparate impact doctrine's business necessity analysis is designed to adjudicate. In this case, the accuracy cost was insufficient to justify the discriminatory impact, and the less discriminatory alternative was available. Organizations should not assume that the most accurate model is always the appropriate model to deploy.

Discussion Questions

1. Meridian's neighborhood risk score was purchased from a third-party vendor as a commercially available data product. The vendor did not disclose how the score was constructed or which variables it incorporated. What due diligence obligations should institutions have when incorporating third-party data products into credit models? How should contracts with data vendors address disparate impact risk?

2. The OCC's Matters Requiring Attention cited the absence of proxy variable analysis from Meridian's original model validation documentation. Should model validation requirements for ML models be more specific about mandating SHAP-based or comparable proxy variable analysis? What would be lost if such analysis were mandatory?

3. The remediated model has a denial rate for majority-minority neighborhoods that is 1.2 times the rate for majority-white neighborhoods, after controlling for individual creditworthiness factors. The audit team assessed this as legally defensible. Do you agree? At what point does a residual demographic disparity become legally or ethically unacceptable, even if it reflects "genuine" differences in financial profiles that are themselves a legacy of historical discrimination?

4. Meridian's proactive disclosure was well-received by regulators. But consider a scenario where the bank's internal audit had found the same problem and the institution chose not to disclose. What are the legal, regulatory, and reputational risks of non-disclosure? What organizational incentives might lead an institution to suppress internal audit findings, and how should governance structures counteract those incentives?