Case Study: When Models Drift: Real-World Deployment Failures

"A model is a snapshot of the world at the moment it was trained. The world does not stop changing because you've deployed a model." -- Shreya Shankar, "Operationalizing Machine Learning"

Overview

Machine learning models are trained on historical data and deployed into a world that keeps changing. When the world changes but the model does not, the result is drift -- a gradual or sudden degradation of model performance that can produce harmful outcomes. This case study examines three real-world deployment failures caused by different types of drift, analyzing what went wrong, how the failures could have been detected, and what responsible AI monitoring practices would have prevented the harm.

Skills Applied: - Identifying concept drift, data drift, and performance degradation in real-world systems - Connecting model monitoring to responsible AI frameworks - Analyzing the organizational conditions that allow drift to go undetected - Designing monitoring systems that catch drift before it produces harm

Case 1: Zillow's iBuying Catastrophe

What Happened

In 2021, Zillow Offers -- Zillow's program that used an AI-powered pricing model (the "Zestimate") to make instant cash offers on homes -- shut down after losing approximately $881 million. The program was designed to use Zillow's pricing algorithm to identify homes that could be purchased below market value, renovated, and resold at a profit.

The model worked reasonably well during the relatively stable housing market of 2018-2020. But in 2021, the housing market experienced unprecedented volatility: rapid price increases in some markets, supply shortages, shifting demand patterns as remote work changed where people wanted to live, and pandemic-driven disruptions to construction and renovation timelines.

The Drift

Concept drift: The relationship between the features the model used (square footage, location, comparable sales, market trends) and the outcome it predicted (fair market value) changed rapidly. Historical patterns no longer predicted current prices. The model was trained on a world where housing prices changed gradually; it was deployed in a world where prices could shift 10-15% in a quarter.

Data drift: The distribution of homes available for purchase shifted. As the market heated up, Zillow Offers faced more competition from other iBuyers and traditional buyers. The homes available to purchase at algorithmic prices were increasingly those that other buyers had passed on -- properties with hidden issues, unusual characteristics, or inflated asking prices.

Organizational failure: Zillow's leadership set aggressive volume targets for Zillow Offers, pressuring the team to purchase more homes faster. This created an incentive to override the model's confidence thresholds -- approving purchases even when the model's uncertainty was high.

The Harm

Zillow ultimately owned approximately 7,000 homes purchased at prices the market would not support. The company laid off 2,000 employees (25% of its workforce) when the program was shut down. The financial loss was borne by shareholders, but the human cost fell on employees who lost their jobs and on homeowners in neighborhoods where Zillow's bulk ownership and rapid resale distorted local housing markets.

What Monitoring Would Have Caught

A responsible deployment monitoring system would have detected:

• Price prediction error rate increasing over successive months, indicating concept drift
• Model confidence scores declining as the input data diverged from training distribution
• Increasing variance in prediction residuals, signaling that the model was becoming less reliable
• Override frequency increasing -- a signal that human decision-makers were compensating for model limitations rather than flagging them for review

Case 2: COVID-19 and Healthcare Prediction Models

What Happened

During the COVID-19 pandemic, healthcare prediction models worldwide experienced simultaneous, catastrophic drift. Models that predicted hospital readmission, patient deterioration, sepsis risk, and disease progression -- systems that hospitals had relied on for years -- began producing unreliable results.

A landmark 2021 study by Wong et al. analyzed 27 clinical prediction models deployed across 12 hospitals and found that model performance degraded significantly during the pandemic. Sepsis prediction models saw their positive predictive value drop by 18-25%. Readmission models overestimated risk for some conditions and underestimated it for others.

The Drift

Concept drift: The pandemic changed the relationship between clinical features and outcomes. Patients with respiratory symptoms who would have been low-risk pre-COVID were now potentially high-risk. Treatment protocols changed rapidly. The relationship between lab values and clinical outcomes shifted as COVID-19 introduced a novel pathogenic mechanism.

Data drift: Patient populations changed dramatically. Elective procedures were cancelled, removing a large category of patients from hospital populations. Emergency departments saw different case mixes. The demographic profile of hospitalized patients shifted as COVID-19 disproportionately affected certain communities.

Data quality degradation: Documentation practices changed during the crisis. Clinicians under extreme workload pressure documented less thoroughly. Diagnostic codes were applied inconsistently as coding standards for COVID-19 evolved. Data entry errors increased.

The Harm

Models that clinicians relied on for decision support became unreliable during the period when clinical decision-making was most difficult. A sepsis prediction model that generates false negatives (failing to flag patients who are deteriorating) in a pandemic environment can contribute to delayed intervention and preventable deaths. The harm was not abstract -- it affected clinical decisions about real patients during a crisis.

What Monitoring Would Have Caught

• Automated performance tracking comparing model predictions against observed outcomes would have detected the accuracy decline within weeks
• Distribution monitoring would have flagged that the incoming patient population differed significantly from the training population
• Calibration checks would have revealed that the model's probability estimates were no longer well-calibrated to actual event rates
• Alert systems triggered by performance degradation would have notified clinical teams that the model should be used with reduced confidence or temporarily suspended

The deeper lesson: many healthcare AI systems were deployed without ongoing performance monitoring. They were validated at deployment and assumed to remain accurate indefinitely. The pandemic revealed this assumption as dangerous.

Case 3: Amazon's Recruiting Algorithm

What Happened

In 2018, Reuters reported that Amazon had developed and then abandoned an AI-powered recruiting tool that showed systematic bias against women. The tool was designed to review resumes and rate candidates on a one-to-five star scale. It was trained on ten years of Amazon's historical hiring data -- data that reflected a male-dominated technology industry.

The Drift (or Absence of It)

This case is technically not about drift in the traditional sense -- the model did not degrade over time. Instead, it faithfully reproduced the biases embedded in its training data. But the case illustrates a related failure: the absence of monitoring for fairness, which allowed biased outputs to persist unchecked.

Historical bias baked in: The training data reflected ten years of hiring in which men were disproportionately selected. The model learned that resumes containing indicators associated with women (women's colleges, women's organizations) were correlated with lower hiring outcomes -- not because women were less qualified, but because the historical hiring process was biased.

Proxy features: The model penalized resumes that contained the word "women's" (as in "women's chess club captain" or "women's lacrosse team") and downgraded graduates of two all-women's colleges. These were proxy features that correlated with gender without explicitly using gender as a variable.

No fairness monitoring: The model was deployed internally without disaggregated performance analysis across gender groups. When the bias was discovered, it was through ad hoc review, not systematic monitoring.

The Harm

The model was used internally for an undisclosed period before the bias was identified and the project was abandoned. During that period, women's resumes were systematically downranked, potentially affecting hiring decisions. The harm extended beyond individual candidates: it demonstrated that even sophisticated companies can deploy biased AI systems when monitoring for fairness is absent.

What Monitoring Would Have Caught

• Disaggregated output analysis by gender would have immediately revealed the systematic difference in candidate rankings
• Feature importance analysis would have identified that features correlated with gender (women's colleges, gendered terminology) were driving the rankings
• A model card documenting the training data would have flagged that ten years of male-dominated hiring data was likely to encode gender bias
• A datasheet for the training data would have documented the gender imbalance in historical hires, alerting developers to the risk before training

Cross-Case Analysis

The Common Thread

All three cases share a common failure: models were deployed and assumed to continue performing acceptably without ongoing monitoring. The specific mechanisms differed -- market volatility, a pandemic, historical bias -- but the organizational failure was the same: the absence of systematic, ongoing evaluation of model performance, fairness, and relevance.

The Monitoring Gap

The Organizational Pattern

In each case, the organization:

1. Invested heavily in model development -- building sophisticated systems with substantial engineering resources
2. Underinvested in monitoring -- deploying without systematic, ongoing performance evaluation
3. Detected failure reactively -- through financial losses, external studies, or ad hoc review rather than through monitoring systems
4. Responded after significant harm -- by which point the damage (financial, clinical, discriminatory) had already occurred

This pattern reveals a systemic bias in AI development: organizations invest in building models but not in watching them. The model card and monitoring frameworks from Chapter 29 are designed to close this gap.

Discussion Questions

1. The incentive problem. Zillow's leadership set aggressive volume targets that pressured the team to override model uncertainty. How should responsible AI monitoring interact with business incentives? What happens when the model says "stop" but the business says "go"?

2. The pandemic stress test. COVID-19 caused simultaneous drift in healthcare models worldwide. Should healthcare AI systems include automatic "circuit breakers" that reduce model authority or require additional human oversight when performance degrades? What would such a system look like?

3. Bias vs. drift. Amazon's case is not traditional drift -- it's bias baked into the training data. But the organizational failure is similar: absence of monitoring. Is the distinction between bias and drift analytically useful, or should monitoring systems treat them as a single category of "model unreliability"?

4. The model card connection. For each of the three cases, describe what a well-constructed model card would have documented at deployment that could have prevented or reduced the harm. Be specific about which model card sections are most relevant.

5. Ongoing monitoring as ethical obligation. Should organizations have a legal obligation to monitor deployed AI systems, with consequences for failures to detect and address drift? How would such a requirement be enforced?

Your Turn: Mini-Project

Option A: Drift Detection Design. Design a monitoring system for one of the three cases. Specify: what metrics are tracked, how frequently, what thresholds trigger alerts, what actions are required when alerts fire, and who is responsible for responding.

Option B: Model Card Retrospective. Write the model card that should have existed at deployment for one of the three cases. Fill in all sections based on publicly available information, including the limitations and ethical considerations that the original deployment failed to document.

Option C: Circuit Breaker Design. Design a "circuit breaker" system for healthcare prediction models that automatically adjusts model authority when performance degrades. Specify: what triggers the circuit breaker, what levels of reduced authority exist (advisory only, suspended, human-only), and what process restores full model authority.

References

• Parker, Will, and Nicole Friedman. "Zillow Quits Home-Flipping Business, Cites Inability to Forecast Prices." The Wall Street Journal, November 2, 2021.

• Wong, Andrew, Erkin Otles, John P. Donnelly, et al. "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients." JAMA Internal Medicine 181, no. 8 (2021): 1065-1070.

• Dastin, Jeffrey. "Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women." Reuters, October 10, 2018.

• Shankar, Shreya, Rolando Garcia, Joseph M. Hellerstein, and Aditya G. Parameswaran. "Operationalizing Machine Learning: An Interview Study." arXiv preprint arXiv:2209.09125, 2022.

• Sculley, D., Gary Holt, Daniel Golovin, et al. "Hidden Technical Debt in Machine Learning Systems." In Advances in Neural Information Processing Systems 28, 2015.

• Klaise, Janis, Arnaud Van Looveren, Giovanni Vacanti, and Alexandru Coca. "Monitoring Machine Learning Models in Production." arXiv preprint arXiv:2007.06299, 2020.

• Rabanser, Stephan, Stephan Gunnemann, and Zachary Lipton. "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift." Advances in Neural Information Processing Systems 32, 2019.