Case Study 1: Gender Shades — When AI Couldn't See Dark-Skinned Women

Background

In 2015, Joy Buolamwini was a graduate student at the MIT Media Lab working on a project she called the "Aspire Mirror" — a system that would project inspiring images onto a user's face using facial detection software. There was one problem: the software could not detect her face. Buolamwini, a Black woman of Ghanaian descent, discovered that the system would only recognize her when she put on a white mask.

This was not an isolated glitch. It was a pattern. And Buolamwini — who went on to found the Algorithmic Justice League — turned that pattern into one of the most influential studies in the history of AI fairness research.

The Study

Working with Timnit Gebru, then a researcher at Microsoft, Buolamwini designed a rigorous evaluation of commercial facial analysis systems from three major technology companies: IBM, Microsoft, and Face++ (a Chinese AI company). Their study, published in 2018 under the title "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification," tested these systems' ability to classify the gender of faces in photographs.

The key innovation was the dataset. Existing benchmark datasets for facial recognition were heavily skewed toward lighter-skinned subjects. Buolamwini and Gebru created a new dataset — the Pilot Parliaments Benchmark — composed of 1,270 photographs of members of parliament from three African nations (Rwanda, Senegal, South Africa) and three European nations (Iceland, Finland, Sweden). The dataset was deliberately balanced for gender and skin tone, using the Fitzpatrick skin type scale (a dermatological classification system) to ensure representation across the spectrum.

The Results

The findings were dramatic:

The pattern was consistent across all three systems: performance was best for lighter-skinned males and worst for darker-skinned females. For IBM's system, the error rate for darker-skinned women was 34.7% — meaning the system got the gender wrong for more than one in three darker-skinned women. That same system achieved a 0.0% error rate for lighter-skinned men.

If the companies had reported only overall accuracy (as was standard practice), the disparities would have been invisible. Overall accuracy rates for all three systems were above 85%. The bias only became apparent when performance was disaggregated — broken down by the intersection of gender and skin tone.

Why It Happened

The root cause was representation bias. The datasets used to train these commercial systems overwhelmingly contained lighter-skinned faces. Darker-skinned individuals, and especially darker-skinned women, were dramatically underrepresented in the training data.

This was compounded by benchmark bias. The standard datasets used to evaluate facial recognition systems — datasets like Labeled Faces in the Wild (LFW) — were similarly skewed. A system could achieve a high score on the standard benchmarks while performing terribly on faces that the benchmarks did not adequately represent. The systems appeared to work well because they were being tested on the same narrow population they had been trained on.

The Response

The Gender Shades study provoked significant responses from the companies involved:

IBM published a response blog post, improved its system, and later — in June 2020 — announced it would exit the facial recognition market entirely, citing concerns about racial justice and mass surveillance. CEO Arvind Krishna wrote to Congress that IBM "firmly opposes and will not condone uses of any technology, including facial recognition technology offered by other vendors, for mass surveillance, racial profiling, violations of basic human rights and freedoms."

Microsoft improved its systems and became a public advocate for facial recognition regulation. Microsoft president Brad Smith called for government regulation of facial recognition technology, arguing that the market alone could not solve the problem.

Face++ initially disputed the methodology but later acknowledged and addressed performance gaps.

After the original study, Buolamwini and colleagues conducted a follow-up audit ("Actionable Auditing," 2019) and found that all three companies had significantly improved their systems — particularly for darker-skinned female faces. The study demonstrated that external auditing works: shining a light on disparities can drive concrete improvement.

The Broader Implications

The Gender Shades study matters beyond facial recognition. It established several principles that apply to AI bias broadly:

1. Overall accuracy is not enough. Performance must be evaluated across demographic subgroups. A system that is 95% accurate overall may be 99.9% accurate for one group and 65% accurate for another.

2. The people most affected are often the least represented. The communities most likely to be harmed by facial recognition in law enforcement — communities of color — were the same communities whose faces were least represented in training data. The people who bore the greatest risk had the least voice in the system's design.

3. Intersectionality matters. The largest disparities were not found by looking at gender alone or skin tone alone, but at their intersection. Darker-skinned women — who sit at the intersection of two axes of marginalization — experienced the worst outcomes. This echoes legal scholar Kimberle Crenshaw's concept of intersectionality: you cannot understand discrimination by looking at one axis at a time.

4. Independent auditing is essential. The companies did not discover these disparities on their own. It took an external researcher — one who, crucially, had personal experience with the problem — to expose them. This underscores the need for independent evaluation, diverse research teams, and audit requirements.

5. Sunlight is a disinfectant. Once the disparities were publicly documented, the companies acted. This suggests that transparency and accountability mechanisms — not just good intentions — drive improvement.

Discussion Questions

1. Why do you think none of the three companies discovered the performance disparities before Buolamwini and Gebru's study? What does this tell us about the limitations of internal testing?

2. The study used members of parliament as its dataset — public figures whose images are freely available. What are the ethical advantages of this dataset choice compared to, say, scraping social media photos?

3. IBM ultimately exited the facial recognition market. Microsoft called for regulation. Both are reasonable responses to the same problem. Which approach do you think is more likely to reduce harm, and why?

4. The study focused on gender classification — a relatively simple task compared to face identification. If bias exists in gender classification, what might you infer about bias in more complex facial recognition applications (like identifying suspects from surveillance footage)?

5. How does the concept of intersectionality change the way we should audit AI systems? What does it mean practically for bias testing?

Connection to Chapter Themes

This case study illustrates several concepts from Chapter 9:

• Representation bias (Section 9.2): The training data did not adequately represent darker-skinned faces
• Measurement bias (Section 9.2): Standard benchmarks measured performance on an unrepresentative population
• The importance of disaggregated metrics (Section 9.3): Overall accuracy masked severe disparities
• Organizational approaches to mitigation (Section 9.5): Independent auditing, diverse research teams, and public accountability drove improvement
• The power question (Section 9.6): The communities most affected by facial recognition had the least input into its development