Case Study 7.2: Scored by a Machine — Credit, Insurance, and Algorithmic Sorting

High-Stakes Classification in Financial Services

The Setup

When Maria Chen applied for a mortgage in 2022, she had a stable income, minimal debt, and had never missed a bill payment. She was denied. The lender's explanation was a single sentence: "Based on our automated assessment, your application does not meet our current lending criteria."

Maria's experience isn't unusual. By the mid-2020s, the majority of consumer lending decisions in the United States involved some form of algorithmic assessment — from traditional credit scores like FICO to newer AI-driven models that incorporate hundreds of data points beyond payment history. These systems classify applicants into risk categories, set interest rates, determine insurance premiums, and decide who gets access to financial products.

The shift from human loan officers to algorithmic classification has been driven by two promises: speed and consistency. An algorithm can process thousands of applications per hour, and — in theory — it treats everyone the same. No more biased loan officers who approve white applicants and deny Black ones for the same financial profile. The algorithm looks at the numbers, not the person.

But "looking at the numbers" turned out to be more complicated than it sounded.

How Credit Scoring Works

Traditional credit scoring (like the FICO score used in the United States) is a classification system that works roughly like this:

  1. Data collection: Credit bureaus gather data about your financial behavior — how much you owe, whether you pay on time, how long you've had credit, what types of credit you use, how often you apply for new credit.

  2. Model training: The scoring model was developed using historical data about borrowers who did and didn't default. It identified patterns: people who missed payments were more likely to default; people with diverse credit types were less likely to default; etc.

  3. Classification: Your data is processed through the model, which produces a score (typically 300 to 850). That score classifies you along a risk spectrum: below 580 is "poor," 580-669 is "fair," 670-739 is "good," 740-799 is "very good," and 800+ is "exceptional."

  4. Decision: Lenders use the score (often in combination with other factors) to decide whether to approve your application and at what interest rate.

This system is relatively transparent — you can look up the factors that influence your score, and the general methodology is publicly documented. The FICO model is also moderately interpretable: it can tell you that your score was lowered by a recent hard inquiry or a high utilization ratio.

The Shift to AI-Driven Models

Newer lending systems go far beyond traditional credit scores. They use machine learning models that process hundreds of variables — not just financial history, but also:

  • Geographic data: Where you live, how long you've lived there, neighborhood characteristics
  • Employment data: Where you work, your job title, your employer's financial health
  • Educational data: Where you went to school, what degree you earned
  • Digital footprint data: Online purchasing behavior, social media activity, device type, browsing patterns
  • Behavioral data: How quickly you fill out an application, whether you read the terms and conditions, what time of day you apply

These models are more accurate than traditional credit scoring in a narrow technical sense: they predict default rates with greater precision. But they raise serious concerns about fairness, transparency, and the meaning of "creditworthiness" itself.

The Proxy Variable Problem

Consider geographic data. An AI lending model that uses zip code as a variable will, in the United States, effectively incorporate information about race — because residential segregation means zip codes are strongly correlated with racial demographics. The model isn't "racist" in the sense of intentionally discriminating. But if it charges higher interest rates to applicants from predominantly Black zip codes, the effect is racial discrimination regardless of intent.

This isn't hypothetical. In 2022, the Consumer Financial Protection Bureau (CFPB) published a report showing that algorithmic lending models produced significant disparities in approval rates and interest rates across racial groups — even after controlling for income, debt, and credit history. The disparities were driven by variables that served as proxies for race: zip code, educational institution, and employment type.

The proxy variable problem extends beyond race. An insurance company that uses credit scores to set premiums is, in effect, charging more to people who have experienced financial hardship — which correlates with disability, chronic illness, divorce, and other life disruptions. A car insurance model that uses occupation may charge more to gig workers, who are disproportionately immigrants and minorities.

The Interpretability Gap

When Maria Chen was denied her mortgage, she received no meaningful explanation. This is increasingly common with AI-driven lending decisions. Unlike the FICO score — which can tell you "your utilization ratio is too high" — a deep learning model that processes 200 variables through multiple neural network layers often cannot produce a human-understandable explanation for its decision.

This creates a practical problem: how do you fix something if you don't know what's wrong? If the system says your utilization is too high, you can pay down your balance. If the system just says "denied," you're stuck.

It also creates a regulatory problem. In the United States, the Equal Credit Opportunity Act requires lenders to provide specific reasons for adverse decisions. The Fair Credit Reporting Act gives consumers the right to know what information was used. But these laws were written for an era of interpretable decision rules, not opaque neural networks. A model that denies an application based on the complex interaction of 200 variables may technically comply with these laws by listing the top contributing factors, even if those factors don't tell the full story.

The Feedback Loop in Lending

Credit scoring creates a classic feedback loop:

  1. People with high credit scores receive better loan terms (lower interest rates, higher limits)
  2. Better terms make it easier to manage debt, building credit history
  3. Improved credit history leads to even higher scores
  4. Higher scores lead to even better terms

The reverse is also true:

  1. People with low scores receive worse terms or are denied credit entirely
  2. Lack of access to credit makes it harder to build a credit history
  3. Thin or negative credit history leads to lower scores
  4. Lower scores lead to further exclusion

This loop is self-reinforcing. It means that your initial credit classification — which may have been influenced by factors beyond your control, including systemic inequities — tends to perpetuate itself. People who start with advantages accumulate more. People who start with disadvantages fall further behind.

The AI-driven expansion of variables makes this feedback loop more powerful. If the model considers your neighborhood, your education, and your employer — and if those factors are themselves products of historical inequality — then the model encodes inequality into credit decisions, which shapes financial outcomes, which feeds back as data confirming the model's predictions.

What's Being Done

Several responses to these concerns have emerged:

  • Regulatory scrutiny: The CFPB under the Biden administration increased oversight of algorithmic lending, requiring lenders to demonstrate that their models don't produce discriminatory outcomes. The EU's AI Act classifies credit scoring as a "high-risk" AI application subject to special requirements.

  • Alternative data approaches: Some fintech companies use alternative data (like rent payments, utility bills, and subscription payments) to evaluate people with thin credit files, arguing this is more inclusive than traditional credit scoring. However, alternative data also raises proxy variable concerns.

  • Explainable AI requirements: Some jurisdictions are beginning to require that AI lending decisions be explainable — that applicants receive not just a decision but a meaningful explanation of why.

  • Fairness testing: Companies like Zest AI and others have developed tools to test lending models for disparate impact across demographic groups, adjusting models that produce inequitable outcomes.

Discussion Questions

  1. The accuracy-fairness tension. An AI lending model is more accurate at predicting defaults than a traditional credit score, but it also produces larger racial disparities. Should a lender be required to sacrifice accuracy for fairness? Who decides what level of accuracy trade-off is acceptable?

  2. The right to explanation. Should everyone who is denied credit (or charged a higher rate) have the right to a detailed, understandable explanation? What would "understandable" mean in the context of a complex AI model? Is a simplified explanation (like the ones produced by LIME or SHAP) sufficient?

  3. Alternative data dilemma. Using rent payments to evaluate thin-file borrowers can expand access to credit. But it also means that a late rent payment (which could result from a landlord dispute, not financial irresponsibility) might damage your credit. On balance, does alternative data inclusion help or hurt vulnerable populations?

  4. Structural vs. individual. Maria Chen's denial was the result of an algorithmic decision, but the underlying patterns the algorithm detected — geographic segregation, educational stratification, wealth inequality — are structural problems that no algorithm can solve. How much responsibility should an AI system bear for reflecting structural inequality? What's the difference between a system that reflects inequality and one that amplifies it?

  5. Connecting to the chapter. Identify where this case study illustrates each of the following concepts: classification, proxy variables, the accuracy-interpretability trade-off, feedback loops, and the distinction between prediction and explanation.

Key Takeaway

Algorithmic credit scoring illustrates how AI classification systems can appear objective while embedding and amplifying the inequities present in their training data and input variables. The shift from interpretable, rule-based systems to complex AI models has improved prediction accuracy while reducing transparency, accountability, and the ability of individuals to understand — and contest — the decisions that shape their financial lives. The question is not whether to use AI in lending, but under what constraints, with what safeguards, and with what accountability to the people whose lives are sorted by the scores.