Case Study 8.2: When the Diagnostic AI Missed the Diagnosis

Medical AI Errors and Their Consequences

The Setup

In early 2023, a 52-year-old woman — we'll call her Elena — visited her primary care physician with persistent fatigue, unexplained weight loss, and occasional shortness of breath. Her physician, who practiced at a mid-sized hospital in the Midwest, had recently gained access to a new AI diagnostic support tool — similar in concept to MedAssist AI, the system we've been following throughout this book.

The AI tool analyzed Elena's symptoms, lab results, and chest X-ray, and returned its assessment: "86% probability: iron-deficiency anemia. Recommend: ferritin panel, iron supplementation, follow-up in 6 weeks." The recommendation was clinically reasonable. Elena's blood work showed slightly low hemoglobin, and her symptoms were consistent with anemia. The physician prescribed iron supplements and scheduled a follow-up.

Six weeks later, Elena returned. She was worse. The iron supplements hadn't helped. Additional testing revealed what the AI had missed: Elena had early-stage lung cancer. The tumor was visible on her original chest X-ray — a subtle shadow that the AI had not flagged. A radiologist, reviewing the X-ray after the cancer diagnosis, said the shadow was "at the boundary of what I'd call suspicious — I might have caught it, I might not."

Elena's cancer was treatable, and her prognosis was ultimately good. But the six-week delay caused anxiety, additional medical visits, and the unsettling knowledge that an AI system she was told was "state of the art" had directed her care away from the actual diagnosis.

Understanding What Happened

Elena's case illustrates multiple failure types working together:

Distributional shift. The AI diagnostic tool had been trained primarily on data from large academic medical centers with modern imaging equipment. Elena's hospital used older equipment that produced images with slightly different characteristics — different contrast, different resolution, different noise patterns. The tumor, which might have been more clearly visible on higher-quality imaging, was near the boundary of what the AI had learned to flag.

Overconfidence. The AI reported "86% probability: iron-deficiency anemia" — a single, specific diagnosis presented with a confidence score that suggested reliability. But the system was poorly calibrated in this context: its confidence scores, validated against data from the training distribution, were less meaningful when applied to images from different equipment. The 86% figure gave Elena's physician a specific number to anchor on, creating a false sense of diagnostic certainty.

Anchoring and automation bias. The physician received the AI's assessment before conducting her own independent evaluation. This created an anchoring effect — her subsequent reasoning was influenced by the AI's initial assessment. Research in cognitive psychology has consistently shown that initial estimates, even arbitrary ones, disproportionately influence subsequent judgments. When the AI said "anemia," the physician's attention was drawn toward evidence consistent with anemia and away from other possibilities.

The narrow framing problem. The AI was asked to classify Elena's symptoms and imaging into diagnostic categories. It returned a single ranked list of possibilities. What it didn't do — and what no current diagnostic AI reliably does — was express meta-level uncertainty. It didn't say: "I see something on this X-ray that I'm not sure about. It might be nothing, but it warrants further investigation." It didn't say: "My confidence in this specific diagnosis is moderate; here are alternative diagnoses that should be actively ruled out." It produced a ranked output that implied a linear progression from most likely to least likely, without flagging genuine ambiguity.

The Broader Pattern

Elena's case is a composite, but the pattern it represents is well documented:

Performance gaps across settings. Multiple studies have found that AI diagnostic tools perform differently across hospitals, patient populations, and equipment types. A 2021 review in Nature Medicine found that most AI diagnostic studies showed significant accuracy drops when evaluated on data from institutions other than those that provided training data. The tools work well in the settings they were designed for. They work less well everywhere else. And "everywhere else" is where most patients receive care.

The accountability gap. When Elena's physician was later asked about the missed diagnosis, she said: "The AI said anemia. The symptoms fit. I followed the recommendation." She wasn't being negligent — she was doing what the tool was designed to help her do. But the question of accountability is unresolved: Did the physician fail by not independently scrutinizing the X-ray? Did the AI vendor fail by not validating the tool at Elena's hospital? Did the hospital fail by deploying a tool without understanding its limitations? Did regulators fail by allowing the tool to be marketed without cross-institution validation requirements?

The answer, uncomfortably, is probably "all of the above, to some degree." But in practice, no single party bears clear responsibility, which means Elena has limited recourse.

The equity dimension. AI diagnostic tools' performance gaps don't fall randomly. They fall along predictable lines: the tools work best at the well-resourced institutions that produced their training data and worst at under-resourced institutions serving more diverse, lower-income, and rural populations. This means that AI diagnostic tools, deployed broadly, could widen health disparities rather than narrow them — improving care where care is already good and degrading it where it's already inadequate.

MedAssist AI: Lessons in Context

Elena's experience maps directly onto the MedAssist AI system we've been tracking:

• MedAssist was trained at top academic centers. Its performance on data from those centers is impressive — it matches experienced specialists. But those centers represent a fraction of where healthcare happens.

• MedAssist reports confidence scores. Those scores are calibrated against the training distribution. When MedAssist moves to a new hospital with different equipment, different patient demographics, and different documentation practices, the confidence scores may no longer mean what they appear to mean.

• MedAssist is positioned as a decision support tool. It's designed to help physicians, not replace them. But "support" is a slippery concept. When a tired physician sees "86% probability: anemia," the "support" functions more like a directive. The tool shapes the decision it was meant to merely inform.

What Adequate Safeguards Would Look Like

Elena's case suggests several safeguards that could reduce the risk of AI diagnostic errors:

Cross-institutional validation. Before deploying an AI diagnostic tool at a new hospital, validate its performance on data from that specific hospital. If performance degrades, recalibrate or restrict the tool's scope.

Transparent confidence calibration. Don't just report confidence scores — report how well those scores are calibrated in the current deployment context. A label that says "This tool's confidence scores have not been validated at this facility" would alert physicians to exercise extra caution.

Alternative diagnosis flagging. Require AI diagnostic tools to explicitly flag alternative diagnoses that should be actively ruled out, rather than presenting only ranked probabilities. "Most likely diagnosis: anemia. However, image features warrant ruling out: pulmonary mass, pleural effusion" would have changed Elena's trajectory.

Workflow integration design. Design the interface so physicians encounter the AI's assessment after forming their own initial impression, not before. This reduces anchoring bias. Some researchers call this "second-opinion mode" — the AI functions as a second opinion rather than a first opinion.

Mandatory ambiguity reporting. Require AI tools to explicitly flag cases where they detect features they can't confidently classify. A system that says "I see something I'm unsure about" is safer than one that stays silent about its uncertainty.

Discussion Questions

1. The anchoring problem. Elena's physician saw the AI's recommendation before conducting her own full evaluation. How does the order of information presentation affect clinical decision-making? Should AI diagnostic tools be designed to present their output only after the physician enters their own initial assessment? What are the trade-offs?

2. Validation and equity. If AI diagnostic tools work best at well-resourced institutions and worst at under-resourced ones, what happens when these tools are deployed broadly? Is there a scenario where broad deployment of a partially effective tool is worse than no deployment at all? Under what conditions?

3. The accountability question. Elena's case involves a physician, an AI vendor, a hospital, and regulators — all of whom contributed to the outcome. How should accountability be distributed? Should AI diagnostic errors be treated differently from ordinary medical errors for liability purposes?

4. Informed consent. Should patients be told when AI tools are being used in their care? Should they have the right to opt out? What would "informed consent" look like for AI-assisted diagnosis?

5. Connecting to the chapter. Identify how each of the following concepts from Chapter 8 appears in Elena's case: distributional shift, overconfidence/poor calibration, automation bias, and the confidence-correctness gap. Which concept do you consider most responsible for the missed diagnosis?

Key Takeaway

Elena's case demonstrates that AI diagnostic errors are not abstract risks — they are concrete harms that affect real patients. The failure wasn't a random malfunction. It was a predictable consequence of deploying a system outside its validated distribution, presenting overconfident assessments to time-pressured physicians, and designing workflows that amplified automation bias rather than mitigating it. The technology worked. The system around the technology didn't. And when AI is deployed in healthcare, the system around the technology is what determines whether patients are helped or harmed.