Case Study 36.2 — The Model That Validated Itself, and the Rule That Followed
Algorithmic decision-making, the explainability requirement, and why a ninety-year-old map is the right place to start
Background: a risk model, drawn in colored pencil
In the 1930s, the Home Owners' Loan Corporation produced residential security maps for cities across the United States. Chapter 2 §2.3 covers them as history. This case study looks at them as engineering, because the structure is what transfers.
Strip away the era and the medium and describe what those maps actually were:
A RISK MODEL, 1930s
INPUTS PROCESS OUTPUT
──────────────────────────────────── ────────── ──────────────────
age and condition of housing stock ┐
income levels of residents │ a scoring a GRADE, A through D
occupations ├──→ rubric ──→ rendered as a color
RACIAL AND ETHNIC COMPOSITION │ applied by on a map
proximity of disfavored groups ┘ appraisers
──────────────────────────────────── ────────── ──────────────────
│
capital allocation ←──┘
It has every part of a model. Defined inputs. A scoring process. A discrete output. And a decision rule attached to the output: capital flowed toward the higher grades and away from the lowest.
The racial and ethnic criteria were not hidden, inferred, or an artifact of correlated data. They were written into the area descriptions, in words, along with the appraisal manuals of the era that treated the presence of particular groups as a depressing influence on value. The record of this is extensive and public.
The issue: the model was accurate, and that is the problem
Here is the part that gets skipped, and it is the part a modern practitioner most needs.
Follow the loop:
THE SELF-CONFIRMING MODEL
┌──────────────────────────────────────────────────────────┐
│ │
↓ │
a neighborhood is graded D │
↓ │
mortgage credit becomes scarce there │
↓ │
fewer sales, less maintenance, less refinancing, │
less new construction, deferred repair │
↓ │
values stagnate or decline; the housing stock deteriorates │
↓ │
measured outcomes show elevated risk in grade-D areas ───────┘
↑
"THE MODEL WAS RIGHT." And within its own data, it was.
An analyst validating those grades against subsequent outcomes would have found them predictive. The model would have passed a backtest. It was accurate — because it had been used.
That is the failure mode. Not merely that a discriminatory policy was encoded, though it was. The deeper problem is that the consequences of the policy became the evidence for the policy, and there is no amount of statistical rigor applied to the outcome data that detects this. The loop is invisible from inside the data. It is only visible if you ask a question the data cannot answer: where did these outcomes come from?
The effects did not stop when the maps did. Housing wealth is inherited. Neighborhood boundaries are durable. Property tax bases, school funding, transit investment, and the distribution of who has a long credit history and who is credit-invisible all carry forward. Chapter 25 works this as doctrine — disparate treatment, disparate impact, and what the law requires of lenders today.
The modern version, stated carefully
No responsible person claims that a contemporary underwriting model is a redlining map. The claim is narrower, and it is a claim about mechanism.
A model trained on historical lending data learns the consequences of past policy as if they were properties of borrowers. Three specific pathways:
Scarcity reads as risk. Where lending was historically withheld, loans are few, unusual, and thin in the training data. Models are conservative about the unfamiliar. Unfamiliarity that was produced by policy is indistinguishable, to the model, from unfamiliarity that reflects genuine risk.
Product history contaminates performance history. If a category of borrower was historically steered toward more expensive or more fragile products, their performance record reflects the product they were sold, not the capacity they had. A model reads the resulting defaults as a fact about the borrowers.
Proxies do the work that prohibited fields cannot. A model given no race field can still produce racially patterned outcomes through geography, employer, banking institution, school district, purchase history, device, or the specific combination of a dozen weak signals. The model does not know what a prohibited basis is. It knows which columns predicted.
Which produces the single most important technical statement in this chapter:
Removing prohibited-basis fields from a model's inputs does not make the model fair, and it is not evidence that the model is fair. Fairness is a claim about outcomes, tested on outcomes, and accompanied by a search for a less discriminatory alternative that meets the same legitimate business need.
What the law actually requires — and why it lands on explainability
This is where the case study stops being about statistics and becomes about a disclosure.
When a creditor takes adverse action, the Equal Credit Opportunity Act and Regulation B require the applicant to receive a statement of the specific principal reasons for the action — or notice of the right to obtain those reasons. Chapter 25 works the notice mechanics, the timing, and who must receive one. The Fair Credit Reporting Act imposes a separate obligation where a consumer report is used, and residential mortgage applicants have their own credit-score disclosure requirement. These are different notices with different triggers.
The requirement is specific and it is principal. It is about this application.
The Consumer Financial Protection Bureau has addressed algorithmic decision-making against this requirement in published circulars. In 2022 it addressed adverse action notification where decisions are based on complex algorithms, and the substance was direct: creditors must provide the specific principal reasons, and the complexity or opacity of the decisioning technology does not create an exception. In 2023 it addressed the proper use of the Bureau's own sample forms, making the related point that a creditor cannot simply select the closest item from a checklist when the actual reason for the decision is not on it.
(Verify the current circulars, their numbering, and any subsequent guidance at the source. Regulatory guidance is revised, withdrawn, and reissued, and a citation in a textbook is a starting point, not an authority.)
In April 2023, four federal agencies — the CFPB, the Civil Rights Division of the Department of Justice, the Equal Employment Opportunity Commission, and the Federal Trade Commission — jointly stated that existing legal authorities apply to automated systems, and that there is no exemption from those authorities because a decision was produced by an algorithm.
Separately, and predating all of it, the federal banking agencies published supervisory guidance on model risk management in 2011, establishing the framework institutions are examined against: disciplined development, implementation, and use; independent validation by people who did not build the model; and governance — policies, a model inventory, documentation sufficient for a knowledgeable outsider to reconstruct the logic, and ongoing monitoring. The guidance was written for banks; non-bank lenders are commonly held to the same concepts in practice.
Put the pieces together and the chain is short:
a decision is made
↓
if adverse, the applicant is owed the SPECIFIC PRINCIPAL REASONS
↓
which requires that someone can STATE what drove THIS decision
↓
which requires the decision to be EXPLAINABLE
↓
─────────────────────────────────────────────────────────────
A DECISION THAT CANNOT BE EXPLAINED CANNOT BE DISCLOSED.
A DECISION THAT CANNOT BE DISCLOSED CANNOT LAWFULLY BE MADE.
─────────────────────────────────────────────────────────────
Explainability is not a research aspiration or a nice property of good software. It is a precondition for using a model in a credit decision at all.
A labeled composite: the model that could not answer the question
This section is a COMPOSITE, constructed to illustrate the governance failure the public guidance describes. No real lender, vendor, model, or product is depicted, and all figures are constructed teaching figures.
A mid-size lender licenses a third-party model that scores incoming applications for likely performance. The vendor's materials describe it as more predictive than the lender's existing approach. The lender proposes to use the score to decline applications below a cutoff.
The compliance officer asks four questions, in this order. The order is the lesson.
"What are the inputs?" The vendor provides a categorical list — credit attributes, application data, and "additional data elements." The additional elements are not enumerated, citing proprietary methods.
Finding: the lender cannot inventory the model. Model risk governance requires an inventory and documentation sufficient to reconstruct the logic. This is a stop, not a caveat.
"For a declined applicant, what are the specific principal reasons?" The vendor's system returns the four highest-contributing factors, expressed as internal feature names.
Finding: better than nothing, but the notice must state reasons in terms an applicant can act on. An internal feature name is not a reason. Someone must be able to translate each one truthfully — and if the translation is a guess, the notice is a guess.
"Which of the inputs could be a proxy for a prohibited basis, and how would we know?" The vendor states that the model does not use race, ethnicity, sex, age, or national origin.
Finding: this answers a question nobody asked. The absence of prohibited-basis fields is not evidence of the absence of disparate impact. The lender must test outcomes and search for less discriminatory alternatives. Chapter 25 owns the doctrine.
"Who validated it, and are they independent of the people who built it?" The vendor's own data science team.
Finding: not independent validation.
The lender does not adopt the model for declines. It adopts it for workflow triage — routing files likely to be complex to more experienced processors — where the output does not determine credit and no adverse action attaches.
That decision is the point of the composite. The model was not rejected because it was a model, or because it was opaque, or because someone was afraid of it. It was placed where its opacity does not create an obligation the lender cannot meet. That is what competent governance looks like, and it is neither hype nor alarmism. It is a matching exercise: the transparency of the tool must be at least as great as the disclosure the decision requires.
What it shows
A model can be accurate and still be wrong. The residential security maps predicted the outcomes they caused. Backtesting cannot detect a feedback loop, because the loop is inside the data.
Historical data is a record of a system, not a record of nature. Every training set in consumer credit is a photograph of who was lent to under prior rules. Chapter 2 explains what those rules were.
The prohibited-basis fields are the easy part. Removing them is trivial and proves nothing. Proxies carry the signal, and the only detection method is outcome testing.
The legal constraint is the disclosure, not the technology. Nothing in ECOA prohibits a sophisticated model. What it requires is that the creditor be able to say why. That requirement sits between the model and the decision, and it will not move.
Governance is a set of questions somebody has to actually ask. Inventory, reasons, proxies, independent validation, ongoing monitoring, and a contract that gets the lender the information it needs to comply. A vendor's assurance is not any of those things.
Outcome
The residential security maps ended as a program. Their effects did not, and the legal architecture that governs lending today — the Fair Housing Act, the Equal Credit Opportunity Act, the Home Mortgage Disclosure Act, and the fair lending examination framework — exists substantially because of what that era demonstrated about what happens when credit allocation is left to unexamined judgment and unexamined models.
The modern outcome is still being written, and honestly so: agencies have stated that existing law applies to automated systems without exception, and the practical burden has landed where it belongs — on the creditor who deploys the model, not the vendor who sold it.
For a loan officer, the outcome that matters is much smaller and entirely within reach. You are the person who says the reason out loud. When you tell a borrower why an application was declined, you are performing the final step of a chain that runs back through a notice, a decision, a model, and a training set. If you cannot complete that sentence honestly, something upstream is broken and you are positioned to be the person who reports it.
The lesson
Chapter 2's maps and Chapter 36's models fail the same way, and the requirement to state a reason is the control that catches both.
A model that cannot be explained cannot produce a lawful denial. That single sentence does more practical work than any amount of discussion about algorithmic fairness in the abstract, because it is enforceable, it is specific, and it applies today.
And note what it protects. It does not protect the lender from a bad model. It protects the applicant — by guaranteeing that somebody, somewhere, has to be able to look them in the eye and name the reason. The maps never had to do that. That is the difference, and it is the whole of it.
Discussion questions
-
Reconstruct the self-confirming loop in four steps. Then explain why a backtest against subsequent outcomes would have shown the residential security maps to be predictive, and what question an analyst would have had to ask instead.
-
A colleague argues that a model cannot discriminate because it never sees race. Rebut this in under 100 words using the word "proxy," and name two variables that could carry the signal.
-
Explain the difference between "the model declined the application" and a statement of specific principal reasons. Then write a sentence a loan officer could truthfully say to a declined borrower in each of two situations: one where the reasons are known, and one where they are not.
-
The composite lender ended up using the model for workflow triage rather than for declines. Explain why that placement resolves the legal problem, and identify one way triage could still create a fair lending issue if implemented carelessly.
-
Model risk management guidance requires independent validation. Define "independent" here, and explain why a vendor's own data science team does not satisfy it even if their work is excellent.
-
Chapter 25 owns disparate impact. Using its framework, describe what a "less discriminatory alternative" analysis would involve for a credit model, and say who in the organization is positioned to do it.
-
A loan officer has no role in building, validating, or approving models. State the five concrete duties §36.9 assigns them anyway, and rank the five by how much protection each provides to a borrower.
-
The chapter says explainability is "a precondition for using a model in a credit decision at all." Argue the strongest counterposition you can construct — then say what it would have to overcome in ECOA, and whether it does.
Sources to go read
- Equal Credit Opportunity Act and Regulation B, particularly the adverse action notification provisions. Free, and shorter than you expect.
- CFPB circulars on adverse action notification and complex algorithms (2022) and on the proper use of the Bureau's sample forms (2023). Verify current numbering and status.
- Joint statement on enforcement efforts against discrimination and bias in automated systems (CFPB, DOJ Civil Rights Division, EEOC, FTC, 2023).
- Supervisory guidance on model risk management (Federal Reserve and OCC, 2011), and any successor or supplementary guidance from your institution's regulator.
- Fair Credit Reporting Act, adverse action and credit-score disclosure provisions applicable to residential mortgage applicants.
- Fair Housing Act and your regulator's fair lending examination procedures.
- The historical record of the Home Owners' Loan Corporation area descriptions, which are archived and publicly accessible. Read the actual language once. It is a more effective teacher than any summary, including this one.