Case Study 1: The Recommendation Engine You Didn't Know About

The System Behind the Score

When Daria Okonkwo applied for car insurance after graduating from college, she expected the process to be straightforward. She had a clean driving record, no accidents, no tickets. She had completed a defensive driving course. She filled out the online application in fifteen minutes and waited.

The quote that came back was $2,400 per year — nearly double what her roommate, Jenna, paid for the same coverage level with the same company. Jenna had one speeding ticket on her record. Daria had none.

What Daria did not know was that the insurance company's pricing system used far more than her driving history to calculate her premium. The AI-driven pricing model — let us call it RateCalc — ingested dozens of data points beyond the obvious ones. It considered her zip code, her credit score, her education level, her occupation, her marital status, and — through third-party data brokers — her consumer behavior patterns: what she bought, where she shopped, and how she used her phone.

[Note: RateCalc is a composite illustration. However, the practices described are well-documented. A 2015 investigation by ProPublica and Consumer Reports found that major insurers charged higher premiums in predominantly minority zip codes, even after controlling for risk factors. A 2020 report by the Consumer Federation of America documented the widespread use of non-driving factors in auto insurance pricing. These are Tier 2 attributions.]

How the System Works

RateCalc is a machine learning model trained on years of claims data. Its goal is to predict, for any given applicant, the likelihood and expected cost of future insurance claims. The model identifies correlations in the data: patterns that statistically associate certain characteristics with higher or lower claims risk.

Some of these correlations make intuitive sense. A driver with multiple accidents is statistically more likely to file future claims. A driver who commutes 60 miles daily has more exposure to risk than one who works from home.

But the model also finds correlations that are statistically valid but ethically troubling. Credit scores, for example, are strong predictors of insurance claims in historical data. People with lower credit scores tend to file more claims. Insurance companies argue this is a legitimate risk factor. Critics argue that credit scores are shaped by systemic economic inequalities — job discrimination, predatory lending, medical debt — and that using them in insurance pricing effectively penalizes people for being disadvantaged.

Here is the key insight: the AI does not know why a correlation exists. It only knows that it exists. RateCalc cannot distinguish between a correlation that reflects genuine risk and one that reflects historical discrimination. It treats all patterns in the training data as equally valid signals.

The Invisible Layer

What makes RateCalc different from the AI systems most people think about — voice assistants, chatbots, image generators — is its invisibility. Daria never saw the algorithm. She never knew which factors drove her quote. The company's website described its pricing as "based on individual risk assessment," which is technically true but practically meaningless.

This invisibility is not unique to insurance. AI-driven pricing and scoring systems operate behind the scenes in:

• Credit decisions. Lenders use AI models to assess creditworthiness, sometimes incorporating non-traditional data like rent payment patterns, social media activity, or even how quickly you scroll through a terms-of-service agreement.
• Hiring. Resume screening tools rank applicants using patterns derived from historical hiring data. If a company historically hired graduates from certain universities, the system may learn to favor those schools — replicating existing advantages.
• Housing. Rental screening algorithms evaluate prospective tenants using credit data, eviction records, and criminal background checks — datasets known to contain errors and to disproportionately affect certain communities.
• Healthcare. A widely cited 2019 study published in Science (Obermeyer et al.) found that a healthcare algorithm used by hospitals across the United States systematically underestimated the health needs of Black patients. The algorithm used healthcare spending as a proxy for health need — but because Black patients historically had less access to healthcare and therefore spent less, the algorithm concluded they were healthier than they actually were.

In each of these cases, an AI system is making or influencing decisions with significant real-world consequences, and the people affected typically have no idea the system exists, no access to its logic, and no meaningful avenue for appeal.

The Power Analysis

To understand why invisible AI systems matter, we need to ask the chapter's central question: Who benefits and who is harmed?

Who benefits from RateCalc: - The insurance company benefits from more granular risk assessment, which can improve profitability. - Some customers benefit from lower premiums if the model identifies them as lower-risk based on non-driving factors. - Shareholders benefit from reduced claims costs.

Who may be harmed: - Customers from lower-income communities, who are more likely to have lower credit scores, may pay higher premiums despite being safe drivers. - Customers in predominantly minority zip codes may face higher rates due to geographic correlations in historical data. - Anyone whose life circumstances — medical debt, job loss, divorce — temporarily lower their credit score may be penalized in ways that have nothing to do with their driving behavior.

The critical point is that the harms are not random. They are patterned. They fall disproportionately on communities that are already economically disadvantaged. And because the system is invisible, the affected individuals cannot identify the source of the disparity, challenge the logic, or advocate for change.

The Asymmetry of Knowledge

There is a structural imbalance at the heart of this story. The insurance company knows everything about Daria — her driving record, her credit score, her shopping habits, her neighborhood demographics. Daria knows almost nothing about the insurance company's decision-making process. She does not know which factors were used, how they were weighted, or whether the model has been audited for discriminatory outcomes. She cannot comparison-shop effectively because competing companies use similarly opaque models with different (and equally hidden) variable weightings.

This asymmetry is not accidental. It is a feature of how these systems are designed and deployed. Companies argue that their pricing models are proprietary trade secrets. Consumer advocates argue that when a proprietary algorithm determines how much someone pays for a necessity like car insurance — which is legally required in most states — the public interest in transparency outweighs the corporate interest in secrecy.

The asymmetry also compounds over time. A young person who receives a high insurance quote because of their zip code or credit score may struggle to pay, miss a payment, see their credit score drop further, and receive an even higher quote the next year. The AI system that was supposed to assess risk can end up creating risk — a pattern we will see in many other contexts throughout this book.

What Could Be Different

This case study is not an argument that AI should never be used in pricing or risk assessment. It is an argument that how these systems are designed, what data they use, and who has oversight matters enormously.

Some reforms that have been proposed or implemented:

• Regulatory limits on input variables. Several U.S. states have banned or restricted the use of credit scores in insurance pricing. These laws reflect a judgment that even if a correlation is statistically valid, using it may be unjust.
• Algorithmic auditing. Independent auditors can test AI pricing models for discriminatory outcomes — checking whether the system produces systematically different results for different demographic groups, even if demographic variables are not directly used as inputs.
• Transparency requirements. Some consumer advocates argue that companies should be required to disclose which factors their AI models consider and how those factors influence outcomes.
• Outcome testing. Rather than focusing on individual input variables, regulators can test whether the outputs of a system produce discriminatory patterns — regardless of which inputs caused them.
• Right to explanation. Emerging regulations in the European Union (particularly the AI Act) establish that individuals affected by high-stakes automated decisions should have a right to a meaningful explanation of how those decisions were made.

Discussion Questions

1. The correlation problem. RateCalc uses credit scores because they statistically predict claims. Is statistical validity sufficient justification for using a data point in a system that affects people's financial lives? Where would you draw the line between a "legitimate" predictor and an "unfair" one?

2. Invisible systems. The chapter argues that most AI is invisible. Should people have a right to know when an AI system is being used to make decisions that affect them? What would meaningful transparency look like in the insurance context?

3. The FACTS Framework applied. Apply the full FACTS Framework to RateCalc. Which question is hardest to answer? What does that difficulty tell you?

4. Structural bias vs. individual bias. RateCalc's designers may have no intent to discriminate. The algorithm itself has no beliefs or attitudes. Yet the system may produce discriminatory outcomes. How should we think about responsibility when harm results from structural patterns in data rather than from individual intent?

5. Reform tradeoffs. Banning credit scores in insurance pricing may result in higher premiums for some low-risk customers who previously benefited from high credit scores. How should policymakers weigh the competing interests of different groups?

Mini-Project

Data Footprint Inventory (30–45 minutes)

For one day, keep a log of every service, platform, or institution that collects data about you. For each entry, note: - What data they likely collect - Whether you consented knowingly - Whether an AI system might use this data to make decisions about you - Whether you could find out how your data is used if you wanted to

At the end of the day, review your log and write a short reflection (200–300 words): Were you surprised by anything? Which data collection practices concern you most, and why?