Case Study 23-2: The Google DeepMind / NHS Data Scandal

Overview

In 2015, Google's artificial intelligence research subsidiary DeepMind entered into a data-sharing agreement with the Royal Free NHS Foundation Trust — one of London's largest NHS hospital trusts — to develop a clinical app called Streams. The app was designed to detect acute kidney injury (AKI) in hospital patients, a genuine clinical need that causes thousands of preventable deaths each year. The partnership seemed like a promising example of AI improving healthcare outcomes.

What made it a scandal — and an enduring case study in healthcare data privacy — was the scope of the data shared. When New Scientist reported on the arrangement in April 2016, it became clear that the Royal Free had provided DeepMind with the personal medical records of approximately 1.6 million patients — not just records relevant to kidney function, but a comprehensive dataset including records related to HIV status, abortions, mental health treatment, and other deeply sensitive conditions. And none of those 1.6 million patients had been informed, much less asked for consent.

This case study examines the arrangement, the investigation that followed, the damage to public trust, and the implications for healthcare AI development.

Background: The Streams App and AKI Detection

Acute kidney injury is a genuine and serious public health problem. It affects approximately 100,000 hospital patients in the UK each year and contributes to around 40,000 deaths — many of which are potentially preventable with timely clinical intervention. The challenge is that AKI can develop rapidly and be missed if clinicians are not monitoring specific biomarkers. An automated alert system that identifies patients at risk of AKI and notifies clinicians could meaningfully save lives.

This was the genuine clinical value proposition of Streams. The app, developed by DeepMind in partnership with the Royal Free, would analyze patients' blood test results against prior results to identify the rapid deterioration characteristic of AKI, and alert the relevant clinical team immediately via a mobile app. Early clinical evaluations suggested the app was effective at identifying at-risk patients and prompting timely intervention.

The clinical rationale was sound. The ethical execution was not.

The 2015 agreement between DeepMind and the Royal Free authorized the transfer of patient data — specifically described as data necessary to support the AKI detection function. What "necessary" meant in practice was interpreted expansively.

The dataset transferred to DeepMind included:

The records of approximately 1.6 million patients who had been treated at three Royal Free hospitals (Royal Free, Barnet, and Chase Farm)
Five years of historical patient data
Data covering not only kidney-related diagnoses but all diagnoses, medications, and treatment records
Records including HIV status, mental health treatment history, records of abortions, and records of drug and alcohol treatment
Patient identifiers including names, dates of birth, and NHS numbers

The rationale for this scope was apparently that AKI detection requires comprehensive patient history — knowing a patient's prior kidney function, medications, and other conditions is clinically relevant to assessing AKI risk. But the scope went well beyond what could be justified by that rationale. A patient whose only contact with the Royal Free was treatment for a mental health condition had no clinical connection to an AKI detection project. Their records were included nonetheless.

None of the 1.6 million patients were notified that their records had been shared with Google DeepMind. None were asked for consent. Most had no reason to believe their records had left NHS systems at all.

The Regulatory Investigation

When New Scientist's reporting brought the arrangement to public attention in April 2016, the Information Commissioner's Office launched an investigation. The ICO's investigation focused on the legal basis for the data transfer — specifically, whether the Royal Free had processed patient data lawfully in sharing it with DeepMind.

The ICO's Findings

The ICO's July 2017 ruling found that the Royal Free NHS Foundation Trust had failed to comply with the Data Protection Act 1998. The finding centered on:

Scope mismatch. The data shared with DeepMind was far broader than what was necessary to support the AKI detection function. The Royal Free had failed to apply any meaningful proportionality analysis to determine what data was actually necessary. Including the records of patients with no kidney-related history, and including the full medical histories of patients whose records were included, violated the principle of data minimization.

Inadequate patient information. Patients had not been informed that their data would be shared with Google DeepMind. The Royal Free argued that patients implicitly consented to their data being used for direct care purposes when they sought treatment. The ICO found that sharing data with an external commercial company for the development of an app — even a clinically beneficial app — could not be characterized as a direct care activity covered by that implied consent.

The "direct care" exception. UK law (following common law and statutory provisions) permits the use of patient data without explicit consent for "direct care" purposes — the treatment of the patient themselves. The Royal Free argued that the Streams app would be used for direct care of the patients whose data it processed. The ICO found that this argument could not justify the use of historical data from 1.6 million patients, most of whom would never be patients of the Royal Free again, to train an algorithm for future patients.

Commercial partnership inadequately characterized. The arrangement with DeepMind was a commercial partnership, not merely a data processing agreement. The Royal Free and DeepMind stood to benefit commercially from the resulting application. The ICO found that this commercial dimension had not been adequately disclosed to patients or factored into the lawfulness analysis.

The ICO issued the Royal Free with a formal undertaking — a binding commitment to comply with specific remedial requirements. The ICO did not impose a financial penalty, which it had the discretion to do. The Royal Free agreed to implement a range of improvements, including conducting a privacy impact assessment for future data sharing arrangements and improving patient transparency.

Subsequent Developments

The ICO investigation focused on the Royal Free rather than DeepMind, because the Trust was the data controller — the entity legally responsible for the patient data. DeepMind was acting as a data processor, subject to the controller's instructions and ultimately the controller's legal obligations. This allocation of responsibility attracted criticism: the more powerful party in the arrangement — Google's DeepMind, with its resources, expertise, and commercial interests — was not the subject of regulatory action.

A follow-up independent review of the arrangement, commissioned by DeepMind itself and conducted by Dr. Julia Powles and Dr. Hal Hodson in 2017, was more critical than the ICO ruling. The review found that DeepMind's initial public statements about the arrangement had understated the scope of data sharing, that the basis for the initial agreement was inadequate, and that the commercial nature of the partnership had been obscured by framing it primarily as a research and direct care project.

The New Patient Data Framework that DeepMind subsequently committed to — including greater transparency, independent oversight, and explicit patient rights — represented an acknowledgment that the initial arrangement had been inadequate. But the commitment was voluntary and untested by regulatory enforcement.

Public Trust Damage

The DeepMind/NHS case illustrates a harm that is difficult to quantify but deeply real: damage to public trust in the use of health data for AI development.

Healthcare AI has enormous legitimate potential. AI systems that identify patients at risk of deterioration, that accelerate diagnosis of complex conditions, that optimize treatment protocols, and that reduce medical errors could prevent enormous amounts of human suffering. But realizing this potential depends on access to high-quality patient data — and access to patient data depends on patients trusting that their data will be used appropriately.

Public surveys conducted after the DeepMind case found that a significant proportion of UK patients were less willing to allow their medical records to be used for research or AI development following the revelations. The case had demonstrated that data they believed was held by their NHS trust could flow to a commercial technology company without their knowledge or consent. Once that trust is damaged, rebuilding it is slow and costly.

The damage was not limited to DeepMind's specific project. It created a broader cloud over health data AI partnerships, making NHS trusts and other healthcare organizations more cautious about data sharing arrangements that might otherwise have served legitimate clinical purposes. The chilling effect on beneficial research is a real cost — one that the people harmed by the lack of timely AKI detection will never be able to trace back to the 2015 data sharing arrangement, but a cost nonetheless.

Implications for Healthcare AI

The DeepMind/NHS case established a set of principles that have since shaped healthcare data partnerships:

Proportionality and Data Minimization

Data sharing for AI development must be proportionate to the clinical purpose. Sharing the records of 1.6 million patients for an app designed to help patients at risk of AKI cannot be justified by the clinical rationale for the app alone. Data minimization — using only the data actually necessary for the purpose — must be applied rigorously, not interpreted as a mere aspiration.

This creates real challenges for AI development, which often benefits from large, comprehensive datasets. The appropriate response is not to abandon data minimization but to develop privacy-preserving alternatives — federated learning, differential privacy, synthetic data — that allow AI development on sensitive data without requiring its transfer to external parties.

Transparency and Patient Information

Patients must be genuinely informed about the use of their data for AI development. This means more than a generic privacy notice buried in a consent form signed at hospital admission. It means proactive, accessible communication that explains who will have access to the data, for what purpose, for how long, and with what protections.

The NHS has since developed a National Data Opt-Out mechanism that allows patients to opt out of their confidential patient information being used for research and planning purposes. The opt-out reflects an acknowledgment that patients have a legitimate interest in controlling how their medical data is used — an interest that the original DeepMind arrangement failed to respect.

Commercial Transparency

When a healthcare organization partners with a commercial company on an AI project, the commercial nature of the arrangement must be transparent to patients, to regulators, and to the public. Framing a commercial development project as a direct care activity obscures the interests at stake and undermines informed consent.

Healthcare data has enormous commercial value. The AI systems built from healthcare data can be proprietary products that generate significant commercial returns. Patients whose data contributes to those returns deserve transparency about how their data is contributing to commercial value, and potentially some form of recognition of that contribution.

Governance Structures

The DeepMind case illustrated the inadequacy of informal governance for commercial healthcare data partnerships. Subsequent frameworks — including NHS England's approach to data access through the Secure Data Environments program — have emphasized structured governance, independent oversight, and accountability mechanisms as prerequisites for data sharing arrangements.

The Researcher vs. Commercial Partner Distinction

Academic researchers accessing patient data for legitimate non-commercial research are typically subject to ethics committee review, data access agreements, and use limitations that reflect the research context. Commercial companies partnering with NHS trusts to develop proprietary products are a different category — but the regulatory frameworks available to address commercial partnerships have been slower to develop than the commercial appetite for healthcare data.

The Broader Context: Healthcare AI and Privacy

The DeepMind/NHS case was not an isolated incident. It reflected a broader pattern in which the commercial opportunity represented by healthcare data has driven data sharing arrangements that outrun the governance frameworks available to ensure those arrangements are legitimate and trustworthy.

Subsequent healthcare data scandals — including the NHS Data sharing arrangement with Palantir, the Google/Ascension "Project Nightingale" in the United States, and various data sharing arrangements between US hospital systems and pharmaceutical companies — have followed similar patterns. Each illustrates the tension between the genuine clinical and commercial value of healthcare data and the legitimate privacy interests of patients who provided that data in the context of seeking care.

The lesson for business professionals is not that healthcare AI partnerships are inherently inappropriate. It is that they require robust governance, genuine transparency, rigorous data minimization, and ongoing accountability — and that the absence of these elements damages not only individual privacy interests but the public trust that makes beneficial healthcare AI possible.

Healthcare AI's potential to save lives is real. Realizing that potential requires building the trust infrastructure that makes patients willing to contribute their data. The DeepMind/NHS case demonstrated how quickly that trust can be damaged — and how difficult it is to rebuild.