Case Study 1: The Algorithm That Deprioritized Black Patients

Background

In 2019, a team of researchers led by Ziad Obermeyer at the University of California, Berkeley, published a study in the journal Science that would become one of the most widely cited examples of algorithmic bias in any domain. The study examined a commercial algorithm used by hospitals and health insurers across the United States to identify patients who would benefit from enrollment in "high-risk care management" programs — intensive, personalized care coordination designed to improve health outcomes for the sickest patients.

The algorithm was not a niche product. It was used to manage the care of approximately 200 million Americans. Its purpose was straightforward and well-intentioned: identify patients who are sick enough to need extra support, and direct resources toward them.

The algorithm was not designed to discriminate. It did not use race as an input variable. Its creators believed they had built a race-neutral tool.

They were wrong.

What the Researchers Found

Obermeyer and his colleagues obtained data from a large academic hospital and analyzed the algorithm's risk scores alongside actual patient health data. They discovered a stark pattern: at any given risk score, Black patients were significantly sicker than white patients.

The numbers were striking. Among patients assigned the same risk score by the algorithm, Black patients had 26.3 percent more chronic conditions than white patients. In practical terms, this meant that the algorithm was systematically assigning lower risk scores to Black patients than their actual health warranted — effectively deprioritizing them for the extra care they needed.

The researchers estimated that fixing this bias would increase the percentage of Black patients identified for extra care from 17.7 percent to 46.5 percent. In other words, more than half of the Black patients who should have received high-risk care management were being missed.

The Mechanism: Cost as a Proxy for Need

How did this happen? The answer lies in a single design choice: the algorithm used healthcare costs as a proxy for healthcare needs.

The logic seemed reasonable on its face. If you want to predict who will need intensive care management in the future, look at who has high healthcare costs now. Patients who are spending a lot on healthcare are, by this logic, patients who are sick.

But healthcare spending is not a neutral measure of health. In the United States, Black patients receive less healthcare spending than white patients at the same level of illness. This disparity arises from multiple, compounding factors:

• Access barriers. Black patients are more likely to be uninsured or underinsured, to live in areas with fewer healthcare providers, and to face transportation barriers to care.
• Implicit bias in treatment. Research has documented that physicians sometimes provide less aggressive treatment to Black patients than to white patients with the same conditions. A landmark study published in the New England Journal of Medicine found that Black patients with cardiovascular disease were less likely to be referred for cardiac catheterization.
• Trust and historical trauma. The legacy of medical experimentation on Black Americans — most notoriously the Tuskegee syphilis study, which lasted from 1932 to 1972 — has understandably eroded trust in the healthcare system for many Black Americans, leading some to defer or avoid seeking care.
• Systemic economic disparities. Income, employment, and insurance status — all of which affect healthcare spending — are shaped by structural racism.

The result: Black patients spend less on healthcare not because they are healthier, but because they face barriers to receiving care. An algorithm that equates spending with need will systematically underestimate the needs of patients who face systemic barriers to spending.

The Response

To their credit, the algorithm's manufacturer worked with Obermeyer's team to understand and address the problem. They tested an alternative approach that used health outcomes (such as the number of chronic conditions) rather than costs as the predictive target. This reduced the bias by approximately 84 percent.

The study prompted a broader conversation across the healthcare industry about the use of proxy variables, the importance of testing for disparate impact, and the limitations of "race-blind" design.

Several major health systems reviewed their use of similar algorithms. The study was cited in congressional hearings on algorithmic bias. It became a standard teaching case in medical schools, public health programs, and computer science departments.

The Deeper Lessons

1. "Race-blind" is not the same as "race-neutral."

The algorithm did not use race as an input. But by using a race-correlated proxy (healthcare costs), it produced race-differentiated outcomes. This is a general principle that extends far beyond healthcare: removing a protected variable from an algorithm does not eliminate bias if the other variables encode the same disparities through correlation.

2. The proxy problem is everywhere.

Healthcare costs are not the only proxy that can encode bias. Credit scores, zip codes, education levels, and employment status are all commonly used variables that correlate with race, class, gender, and other protected characteristics. Any algorithm that relies on such proxies risks reproducing the inequalities they reflect.

3. Aggregate accuracy can hide disparate impact.

The algorithm was accurate on average. It correctly identified many high-risk patients. If the researchers had only looked at overall performance metrics, they might never have discovered the racial disparity. It was only by disaggregating the results — looking at performance separately for Black and white patients — that the problem became visible.

4. Good intentions do not prevent bad outcomes.

The algorithm's designers were not trying to discriminate. They chose cost as a proxy because it was readily available, strongly predictive, and intuitively logical. The bias was not the result of malice but of a design choice made without adequate consideration of its differential effects.

5. The problem is fixable — but only if you look for it.

When Obermeyer's team replaced cost-based predictions with health-based predictions, the bias was substantially reduced. This suggests that the problem is not inherent to using algorithms in healthcare — it is inherent to using algorithms without rigorously testing for equity.

Discussion Questions

1. The Proxy Problem: The algorithm used healthcare costs as a proxy for healthcare needs. In your own words, explain why this proxy was biased. Can you think of a situation in your own life where a proxy measure might not accurately represent what it claims to measure?

2. Race-Blind Design: The algorithm's designers believed that not using race as an input variable made the algorithm fair. Why was this belief mistaken? What does this case teach us about the difference between "race-blind" and "race-neutral"?

3. Systemic vs. Individual Bias: This case involves systemic bias rather than the bias of any individual. No single person decided to deprioritize Black patients. How does systemic bias differ from individual prejudice, and why is it harder to detect and address?

4. The Fix: Obermeyer's team reduced the bias by 84 percent by changing the predictive target from costs to health outcomes. Is an 84 percent reduction sufficient? What should the standard be? Is any algorithmic disparity acceptable, and if so, how much?

5. Scale of Impact: This algorithm affected approximately 200 million Americans. What does this scale tell us about the importance of auditing algorithms for bias? Should there be mandatory equity testing for algorithms used at this scale?

6. Connecting to Your Project: Does the AI system you are auditing for your AI Audit Report use any proxy variables? Could those proxies encode biases similar to those found in the Obermeyer study? How would you test for this?