Case Study 15.2: "Explaining AI to Patients"

Electronic Health Record AI Communication and the Epic Deterioration Index

Overview

Since 2020, Epic Systems — the electronic health records (EHR) company whose platforms are used by more than half of US hospitals — has deployed an AI model called the Deterioration Index (DI) in thousands of clinical settings. The model processes dozens of patient data points in near-real-time and generates a score intended to predict which patients are at elevated risk of clinical deterioration, sepsis, or death. The DI score appears in nurses' EHR dashboards, and in many hospitals, a score above a certain threshold triggers a rapid response team notification or other clinical escalation.

The DI is one of the most widely deployed clinical AI tools in American medicine. Millions of patients have had their care affected by it. The vast majority of those patients have never been told the tool exists, let alone what it calculates, what its accuracy is, or how it influenced the care decisions made about them.

This case study examines the communication gap between AI clinical decision support and the patients and families it affects, using the Epic DI as a central illustration — while drawing on the broader literature on patient communication in AI-augmented healthcare.

How the Deterioration Index Works

The Epic DI is a machine learning model trained on historical EHR data to predict clinical deterioration. It incorporates variables including vital signs (heart rate, respiratory rate, blood pressure, oxygen saturation, temperature), laboratory values (white blood cell count, lactate, creatinine), nursing assessment findings, medication administration records, and other data points routinely captured in the EHR. The model processes these variables continuously as new data is entered and updates the DI score in real time.

Epic has published information about the DI's general methodology but has not disclosed the full list of variables or their weights. The model is trained on de-identified patient data from Epic's network of health system customers, meaning the training data reflects patient populations across many institutions — though individual hospitals may customize how the score thresholds are set and what clinical responses they trigger.

Hospitals that use the DI set their own alert thresholds based on local sensitivity and specificity tradeoffs. A hospital that sets a low DI threshold will generate more alerts (higher sensitivity, catching more true positives) but will also generate more false alarms (lower specificity, burdening nursing staff with alerts for patients who are not deteriorating). Hospitals that set high thresholds will generate fewer alerts, potentially missing some patients who are genuinely deteriorating.

Performance benchmarks for the DI, published in peer-reviewed literature and Epic's own materials, show it outperforms traditional early warning systems in predicting deterioration. But these aggregate metrics obscure significant heterogeneity: the DI's performance varies by hospital setting, patient population, and the clinical circumstances in question.

What Patients Are Not Told

In the typical hospital using the Epic DI, a patient admitted with pneumonia, a post-surgical patient recovering in the ward, or a patient on a medical unit being monitored for potential complications has no knowledge that an AI system is generating predictions about their clinical trajectory. The DI score appears on the nurse's workstation, not in any communication the patient receives. If the DI score rises and a rapid response team is called to the patient's room, the patient may be told "we noticed some changes in your vitals and want to check on you" — without disclosure that an algorithm triggered the response.

This opacity is typical rather than exceptional. A 2021 survey of academic medical centers found that fewer than 10% of hospitals using clinical AI tools had developed patient-facing disclosure policies specifying when and how patients would be informed of AI involvement in their care. In the absence of explicit policy, individual clinicians make ad hoc decisions about what to tell patients — decisions shaped by time constraints, discomfort with AI-related uncertainty, and the general norms of clinical communication, which historically have not required disclosure of specific decision support tools.

Patients who are informed that their care may have been influenced by an AI system — whether they discover this through hospital materials, news coverage, or clinical staff — frequently have questions that clinical teams cannot adequately answer. What does the DI score mean? How accurate is it? If it produces a high alert for a patient who is not actually deteriorating, what happens — and who knows? If it fails to alert for a patient who is deteriorating, is anyone accountable for that failure?

Clinical Staff Communication and AI Transparency

The communication gap between AI tools and patients is mediated by clinical staff — nurses and physicians who receive AI-generated information and must decide how to use it and what to communicate to patients. This mediation creates several distinct failure modes.

Clinicians who cannot explain what they do not understand. Many nurses using the Epic DI have limited understanding of how the model works. Training on the DI in many hospitals is limited to: how to find the score in the EHR, what the alert thresholds mean in terms of expected response, and how to document a DI-triggered response. Training on the model's methodology, its known limitations, its false positive and negative rates, and how to explain it to patients is often absent. A nurse who cannot explain to a patient why a rapid response team was called — beyond "the monitor flagged you" — is not equipped to provide meaningful disclosure.

Clinicians who disagree with AI recommendations. The DI, like all AI clinical tools, sometimes produces outputs that experienced clinicians believe are wrong. A nurse who has cared for a particular patient for a full shift may believe the patient is doing well despite a high DI score — and may be correct. But the organizational and liability structures surrounding AI tools often create pressure to respond to alerts regardless of clinical judgment, precisely because documenting that an alert was received and no action was taken creates a potential liability if the patient subsequently deteriorates. This structural pressure undermines the genuine human oversight that AI clinical tools are supposed to operate within.

Families in crisis. The communication problem is most acute when patients are critically ill and families are present. A family whose loved one is in the ICU, who has been told that the hospital is using an AI system to predict deterioration, may interpret every change in the AI score as a harbinger of their loved one's death — without understanding the base rates, the false positive rates, or the clinical context. Poorly managed AI communication in critical care settings can cause significant distress without providing any useful information.

The ProPublica Investigation and Broader Scrutiny

Investigative journalism and academic research have significantly expanded public understanding of clinical AI tools' limitations. A 2021 investigation by Stat News identified serious performance problems with several clinical AI tools deployed during the COVID-19 pandemic, including tools that had been trained on pre-pandemic data and were being applied in pandemic conditions without revalidation. The investigation found that hospitals often lacked information about whether the AI tools they were using were performing adequately in their patient populations.

Research published in academic journals has raised questions about the DI specifically. Studies examining the DI's performance in real-world hospital settings have found results that vary significantly from the performance reported in Epic's development data — a common pattern with clinical AI models that perform well in training environments and less well when deployed in diverse real-world settings. A 2022 study found that the DI's performance varied significantly by race, with the model generating proportionally different alert rates and different positive predictive values for patients of different racial and ethnic backgrounds.

These findings have not been systematically communicated to patients or families. They are available in medical literature accessible to clinicians and researchers, but they have not shaped the communications patients receive — which, in most hospitals, remain at zero.

Toward Better Patient Communication in Clinical AI

Advocates for improved patient communication in clinical AI have proposed several reforms, drawing on existing informed consent doctrine, shared decision-making frameworks, and the broader AI ethics literature.

General disclosure at admission. Hospitals could include information about clinical AI tools in admission paperwork, alongside existing disclosures about hospital practices. This would inform patients at a high level that AI tools are used to support clinical decision-making — not giving false precision about specific tools or their outputs, but establishing the general context. Surveys of patient preferences have consistently found that patients want to know when AI is being used in their care, even if they do not want detailed technical information.

Clinician training on AI communication. Nurses and physicians should be trained in how to discuss AI tools with patients and families when the topic arises — and increasingly, it does arise, because patients who research their conditions online encounter information about clinical AI. This training should cover: how to describe what the AI tool does in plain language; how to address questions about accuracy and limitations; how to explain the role of human clinical judgment alongside the AI; and how to maintain patient and family trust when the AI is producing unexpected outputs.

Documentation of AI-influenced decisions. Clinical documentation should reflect when AI tools influenced clinical decisions — not because this creates legal exposure, but because it enables internal accountability and improvement. When a rapid response team is called based on a DI alert, the documentation should note that; when a clinician overrides or ignores a DI alert, that should be documented too. This documentation enables quality review: are overrides correlated with better or worse outcomes? Are alert-driven responses correlated with better or worse outcomes? The answers to these questions shape how the tool should be calibrated and used.

Transparency about AI performance data. Hospitals should evaluate AI clinical tools in their own patient populations — not just rely on vendor-reported performance metrics from different populations — and should make this performance information available to clinicians and, in appropriate form, to patients. A hospital that has evaluated its DI implementation and found that it performs differently for certain patient subpopulations has a responsibility to communicate this limitation to the clinical staff using the tool and to consider whether disclosure to affected patients is appropriate.

The most contested question in the clinical AI communication debate is whether use of AI diagnostic or decision support tools requires informed consent from patients. The traditional informed consent doctrine, developed in the context of procedures and treatments, requires clinicians to disclose material information about the risks and benefits of proposed interventions. It is not applied to the tools and methods physicians use in reaching clinical judgments.

The argument for extending informed consent to AI tools is that some clinical AI tools function not merely as diagnostic aids (like laboratory tests) but as inputs to consequential decisions about treatment, with their own error profiles, known biases, and performance limitations that patients have a legitimate interest in knowing. Under this view, using an AI tool to recommend against further cancer treatment — as in this chapter's opening scenario — involves a decision with significant effects on the patient that warrants disclosure and the opportunity for informed patient participation.

The argument against is practical and structural: applying informed consent to every AI tool used in clinical decision-making would be unworkable in a modern hospital environment, where dozens of AI-assisted processes operate simultaneously. A more tractable approach may be general disclosure at the system level (patients are informed that AI is used in clinical care), combined with specific disclosure when AI plays a particularly significant role in a major decision — similar to how patients are informed when a case is discussed at a tumor board, without requiring consent for each piece of data input into the board's deliberations.

Reflection Questions

Should hospitals be required to inform patients when AI systems have been used in their care? If so, what should the disclosure contain, and at what point in the clinical process should it occur?
A nurse believes a patient's DI score is artificially elevated due to a data entry error in the EHR, but the hospital's protocol requires escalation when the score exceeds the threshold. What should the nurse do, and what communication responsibilities does she have to the patient?
Research has found that the Epic DI performs differently across racial and ethnic groups. What obligations does this finding create for hospitals using the tool? For Epic? For the physicians and nurses who rely on the tool's outputs?
A patient's family learns — through a news article — that the hospital used an AI system to help decide that their loved one should be transitioned to palliative care. They are furious that they were not informed. How should the hospital respond, and what should it change about its future practices?
How should clinical AI communication obligations differ between contexts of acute emergency care (where time is short and patient capacity for complex communication may be limited) and elective or chronic care (where there is more time for informed discussion)?