Case Study 2: The Hiring Algorithm That Screened Out Women

Background

In 2014, a team of engineers at Amazon began working on an experimental AI tool that would automate the first stage of the hiring process. The goal was ambitious and, on its face, sensible: Amazon receives hundreds of thousands of job applications each year. A system that could scan resumes and identify the top candidates — rating them on a one-to-five-star scale, like Amazon's own product reviews — would save enormous amounts of time and money.

The team trained their machine learning models on resumes submitted to Amazon over a ten-year period, roughly 2004 to 2014. The system was supposed to learn what a "good" hire looked like by studying the patterns in resumes of people who had been successfully hired.

By 2015, the team realized they had a problem. The system was not rating candidates in a gender-neutral way. It had learned to penalize resumes associated with women.

What Happened

According to reporting by Reuters, which broke the story in October 2018 based on interviews with five people familiar with the project, the system exhibited several forms of gender bias:

Penalizing gender-associated terms. The system downgraded resumes that included the word "women's" — as in "women's chess club captain" or "women's volleyball." These terms correlated with female applicants, and since the historical data skewed male (particularly for technical roles), the system learned to treat them as negative signals.

Downgrading women's colleges. Graduates of two all-women's colleges received lower scores. The system had learned that graduating from one of these institutions was a predictor of not being hired — because historically, fewer graduates of women's colleges had been hired into Amazon's technical roles.

Favoring masculine-coded language. The system reportedly preferred resumes that used language patterns more common in male-written documents — assertive, aggressive phrasing over collaborative or communal language. Research in linguistics and organizational behavior has documented systematic gender differences in resume language, and the AI learned these patterns as quality signals.

Amazon attempted to modify the system to remove the explicit gender penalties, but according to sources interviewed by Reuters, the team could not be confident that the system would not find other proxy signals for gender. The models were complex enough that eliminating one discriminatory pathway did not guarantee others would not emerge.

Amazon disbanded the team and scrapped the project in 2017. A company spokesperson told Reuters that the tool "was never used by Amazon recruiters to evaluate candidates."

Anatomy of the Bias

Let us trace how bias entered this system at each stage of the pipeline:

Problem Formulation

The system was designed to predict "who Amazon should hire" — but it was actually trained to predict "who Amazon did hire." These are different questions. The historical hiring decisions encoded the biases of human recruiters, the cultural norms of the tech industry, and the structural barriers that limited women's participation in technical fields.

Data Collection

The training data — ten years of resumes — reflected a period when the tech industry was overwhelmingly male, particularly in engineering and software development roles. According to various industry reports, women held only about 20–25% of technical roles at major tech companies during this period. The data was not a random sample of the qualified population; it was a record of who made it through a biased pipeline.

Labeling

The "label" — the thing the system was trying to predict — was whether an applicant was hired. But hiring decisions are not objective measures of candidate quality. They are human judgments made by hiring managers who may have conscious or unconscious biases, in an industry with well-documented diversity problems.

Feature Selection

Resume text was used as input features, and the model identified patterns associated with successful hires. Many of those patterns turned out to be proxies for gender: the name of a college, the presence of certain words, patterns in language style. The model was not looking at gender — but it was finding gender everywhere.

Optimization

The model was optimized to match historical hiring decisions as closely as possible. The better it got at this task, the more effectively it reproduced the biases in those decisions. The optimization was working perfectly — just not toward the goal the developers actually intended.

The Deeper Lesson

Amazon's story is not primarily a story about careless engineering. By all accounts, the team that built the system was technically skilled and recognized the problem when it emerged. The lesson is more fundamental: when you train a system to reproduce historical patterns, and those patterns include discrimination, the system will learn to discriminate — often in ways that are difficult to detect and difficult to fix.

Several aspects of this case are worth highlighting:

The bias was not intentional. Nobody wrote code that said "penalize women." The discrimination emerged from the interaction between biased training data, a learning algorithm optimized to match historical patterns, and the deep correlations between gender and other features in resume text.

Removing explicit gender signals was not enough. Even after the team tried to strip gender-associated features, they could not be confident the system was unbiased. This is the "fairness through unawareness" problem in practice: in a world where gender correlates with hundreds of other variables, removing the word "gender" does not remove gender from the model.

The problem was caught internally. Amazon deserves credit for identifying and ultimately killing the project. But the story also raises the question: how many similar systems are in use at other companies, where the bias has not been caught — or where it has been caught but deemed acceptable?

The stakes are high. Hiring decisions are among the most consequential decisions an algorithm can make. They determine people's livelihoods, career trajectories, and economic mobility. Biased hiring algorithms do not just produce unfair scores — they shape who gets opportunities and who does not.

The Broader Landscape

Amazon's story is not isolated. Research has documented similar patterns across the AI hiring industry:

• A 2019 study by researchers at Northeastern University and the University of Southern California found that Facebook's ad delivery system showed job ads to users in ways that reflected gender and racial stereotypes — lumber jobs to men, supermarket cashier jobs to women — even when the advertiser had not specified any demographic targeting.

• Multiple studies have documented that name-based bias (differential treatment based on names associated with different racial or ethnic groups) persists in automated screening systems, even when race is not an explicit input.

• The rapid growth of AI-powered video interview analysis tools — which claim to evaluate candidates based on facial expressions, word choice, and tone of voice — has raised additional concerns about disability discrimination, cultural bias, and the validity of using behavioral cues as proxies for job performance.

The state of Illinois passed a law (the Artificial Intelligence Video Interview Act) in 2019 requiring employers to notify candidates when AI is used to analyze video interviews and to obtain their consent. The European Union's AI Act classifies AI systems used in employment as "high-risk," subjecting them to mandatory requirements for transparency, human oversight, and bias testing.

Discussion Questions

1. Amazon's system was trained on a decade of hiring data. If you were redesigning the system, what alternative training data or labels might you use instead? What challenges would your alternative approach face?

2. The system penalized the word "women's" on resumes. Can you think of other resume features that might serve as gender proxies — even features that seem gender-neutral on the surface?

3. Amazon scrapped the project entirely rather than trying to fix it. Was this the right decision? Can you envision a version of an AI hiring tool that could be built responsibly? What safeguards would it need?

4. The story was reported by Reuters based on interviews with people "familiar with the project." Amazon confirmed some details but has never published a full technical account. Does this matter? What are the implications of the fact that most AI failures in industry are never publicly documented?

5. If you were a job applicant whose resume was scored by such a system, what information would you want the company to disclose? What recourse should you have if you believed the system was biased?

Connection to Chapter Themes

This case study connects to multiple concepts from Chapter 9:

• Historical bias (Section 9.2): The system faithfully learned patterns from a biased history
• Proxy variables (Section 9.1): Gender-neutral features served as proxies for gender
• Fairness through unawareness (Section 9.1): Removing gender did not remove gender bias
• The pipeline (Section 9.1): Bias entered at problem formulation, data collection, labeling, and feature selection — not at any single point
• The "tools built by humans" theme: The AI did not invent gender bias. It inherited and automated it. The discrimination was always human — the AI just made it faster, more consistent, and harder to see