Chapter 17: The Right to Explanation

Part III: Transparency and Explainability

Opening: A Right Without a Remedy?

In May 2019, a Dutch man named Eduard Driessen — or rather, his legal situation — became a minor landmark in European data protection law. Driessen had applied for a bank loan and been refused. He asked the bank to explain why. The bank told him the decision had been made by an automated scoring system. He asked for an explanation of how the system worked. The bank declined, citing proprietary information. He filed a complaint with the Dutch Data Protection Authority, citing Article 22 of the GDPR. The authority investigated. The bank eventually provided a description of the scoring model's general methodology and the categories of factors it considered. It did not provide the weights assigned to those factors, the specific data about Driessen that had driven his particular score, or the specific reasons his score fell below the approval threshold.

Was this adequate under GDPR? The answer — which neither the Dutch Data Protection Authority nor any court has definitively resolved — is: it depends on what "meaningful information about the logic involved" means, and that phrase has been contested since the regulation was adopted.

This is the central puzzle of the right to explanation: the legal formulations that articulate it are broad and compelling in theory but elusive in practice. The right to know why a consequential decision was made about you is deeply rooted in principles of autonomy, dignity, and due process. It is partially expressed in law — most notably in GDPR Article 22 — and is evolving through regulatory guidance, litigation, and legislation in multiple jurisdictions. But the gap between the right as stated and the right as delivered remains vast.

This chapter examines the philosophical foundations of the right to explanation, the legal frameworks through which it has been (incompletely) instantiated, the technical challenges that complicate its implementation, the academic debates about what the right actually requires, and what organizational practices are needed to build genuine explanation capacity. It concludes by examining the global variation in explanation rights — strongest in the EU, developing in the United States through state-level experimentation, and articulated in novel ways in China and other jurisdictions — and what this variation means for organizations operating globally.

Learning Objectives

By the end of this chapter, students will be able to:

1. Articulate the philosophical foundations of the right to explanation in terms of autonomy, dignity, epistemic justice, and democratic accountability.

2. Accurately describe the scope and requirements of GDPR Article 22, distinguishing between what the regulation clearly requires and what remains disputed.

3. Summarize and evaluate the principal positions in the academic debate about whether GDPR creates a genuine right to explanation.

4. Identify the transparency and explanation obligations under the EU AI Act and explain how they interact with GDPR Article 22.

5. Compare the right to explanation in the United States with the EU framework, identifying the primary gaps and the state-level initiatives that are partially filling them.

6. Analyze explanation rights in sector-specific contexts including criminal justice, healthcare, employment, government benefits, and financial services.

7. Evaluate the technical challenges to implementing meaningful AI explanation, including the faithfulness problem, the audience problem, and the gaming problem.

8. Apply the principles of systemic transparency — beyond individual explanation — to AI accountability frameworks.

9. Describe the global variation in explanation rights across the EU, US, China, Canada, and other jurisdictions.

10. Design organizational practices for building explanation capacity at scale.

Section 17.1: The Philosophical Foundation

The right to explanation is not a new idea in law or ethics. It is the application to AI of principles that have been central to liberal legal and political philosophy for centuries. Understanding these foundations helps clarify what the right actually demands and why it matters.

Autonomy

The most fundamental philosophical foundation of the right to explanation is autonomy — the capacity of individuals to direct their own lives through their own choices and reasoning. Autonomy is not merely the absence of constraint; it is the positive capacity to understand one's situation, form views about it, and act on those views. When a consequential decision is made about a person — affecting their access to credit, employment, housing, healthcare, or liberty — and that decision is opaque, the person's autonomy is compromised. They cannot evaluate the decision's basis, cannot form a reasoned view about whether it was correct, and cannot make informed choices about how to respond. The opacity of the decision reduces them to a passive recipient of outcomes they cannot understand or meaningfully contest.

This autonomy argument supports a robust right to explanation: explanation is necessary not merely as a courtesy but as a precondition for the exercise of the autonomous agency that liberal political theory takes as foundational. It is worth noting that this argument applies regardless of whether the decision was correct. A person who was correctly turned down for a loan has the same autonomy interest in understanding why as a person who was incorrectly turned down. The right to explanation is not conditioned on the decision being wrong.

Dignity

Related to but distinct from the autonomy argument is the dignity argument. To be treated with dignity is to be treated as an end in oneself — as a being whose interests and perspectives matter intrinsically — rather than as a means to others' ends, a resource to be processed, or an obstacle to be managed. Being subjected to an opaque algorithmic decision — receiving an outcome without a reason, a verdict without evidence — is a form of dignity violation. It treats the person as a data point to be classified rather than a human being whose situation deserves engagement.

The dignity argument supports not just the provision of explanations but their quality. An explanation that is technically adequate — that satisfies a minimal legal requirement — but is practically unintelligible to its recipient, or that treats the person's situation as an instance of a category rather than engaging with their specific circumstances, may satisfy the legal requirement while falling short of the dignitary standard.

Epistemic Justice

The philosopher Miranda Fricker has introduced the concept of epistemic injustice — the harm done to persons in their capacity as knowers. Relevant here is hermeneutical injustice: the harm that occurs when someone lacks the conceptual resources to understand or articulate their own experiences and circumstances. When AI systems make consequential decisions and the people they affect cannot access meaningful explanations — because the explanations are unavailable, unintelligible, or in a language they do not speak — those people suffer hermeneutical injustice. They are systemically prevented from understanding what is happening to them.

This framing extends the argument for explanation rights beyond individual fairness to a systemic concern about who gets to know things about the systems that govern their lives. Epistemic justice requires not just that explanations be available in principle but that they be accessible to the people who need them — which requires attention to literacy levels, language, disability access, and the organizational processes through which explanations are communicated.

Due Process

In legal and political theory, due process — the requirement that governmental and quasi-governmental actors follow fair procedures before depriving persons of life, liberty, or property — has long included explanation rights. The Fifth and Fourteenth Amendments to the US Constitution protect against deprivation without due process; administrative law doctrine requires agencies to give reasons for their decisions; common law notice requirements have required explaining the basis for adverse actions. The application of algorithmic decision-making to contexts that have historically required reasoned explanation — government benefits, licensing, criminal justice — does not eliminate the explanation requirement. It creates a new technical challenge for fulfilling it.

Historical Context: Administrative Law's Explanation Requirements

Administrative law — the body of rules governing how government agencies make decisions — has required reasoned explanation for agency decisions for decades. The Administrative Procedure Act (1946) requires federal agencies to provide "a statement of the basis and purpose" for rules and regulations, and courts reviewing agency action apply the "arbitrary and capricious" standard to ensure that agencies have genuinely reasoned through their decisions. An agency action that cannot be explained is, almost by definition, arbitrary.

These administrative law explanation requirements were developed in the context of human decision-making, where "the basis and purpose" of a decision could be articulated by the human decision-makers who had made it. They apply with equal force to algorithmic decision-making by agencies — indeed, several courts have expressed concern about whether agencies that rely on algorithmic models can satisfy the reasoned explanation standard if those models cannot be meaningfully explained.

Section 17.2: GDPR Article 22 — What the Law Actually Says

GDPR Article 22 is the most significant legal articulation of the right to explanation in the world. Its text is concise, its scope is contested, and its implementation is profoundly challenging. Understanding what it says — and what it does not say — is essential for any practitioner or researcher in AI ethics.

The Text

Article 22(1) states: "The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her."

Articles 22(2) through 22(4) establish exceptions — where automated decision-making is permitted despite the general rule — and conditions: where automated decision-making is permitted by exception, the controller must "implement suitable measures to safeguard the data subject's rights and freedoms and legitimate interests, at least the right to obtain human intervention on the part of the controller, to express his or her point of view and to contest the decision."

The right to "meaningful information about the logic involved in the automated decision-making" is not in Article 22 itself. It is in Article 13(2)(f) and Article 14(2)(g), which require controllers to provide this information in their privacy notices — in advance of data collection, not upon request. Recital 71 provides additional guidance: data subjects should have the right to explanation of the decision reached after such assessment and to challenge the decision.

"Solely Automated": The Central Interpretive Question

The phrase "solely on automated processing" is the most consequential interpretive question in Article 22. If any human is involved in a decision, does Article 22 not apply? The regulatory guidance from the Article 29 Working Party (now the European Data Protection Board) says no: human involvement that is nominal — rubber-stamping an automated recommendation without genuine assessment — does not take a decision outside Article 22's scope. What is required is meaningful human involvement that can actually influence the decision.

But this guidance, while authoritative in EU regulatory terms, does not have the force of law, and courts in different EU member states may interpret the "solely automated" requirement differently. The question of when a nominally human-reviewed decision is actually "solely automated" for Article 22 purposes is one of the most practically significant unresolved questions in EU AI law. Organizations that structure their human review processes as fig leaves — ensuring nominal human involvement to avoid Article 22 obligations while maintaining effective algorithmic control — are likely to face enforcement attention as regulators become more sophisticated.

"Legal or Similarly Significant Effects": What Decisions Are Covered

Article 22 applies to decisions with "legal effects" or effects that "similarly significantly" affect the data subject. Legal effects include decisions that directly affect legal rights — parole decisions, benefit eligibility determinations, court-ordered monitoring. Similarly significant effects have been interpreted to include credit decisions, insurance pricing, employment decisions, housing decisions, and healthcare decisions — decisions that significantly affect individuals' economic circumstances, opportunities, or wellbeing.

The scope of "similarly significant" is intentionally broad and has been interpreted broadly by most regulatory guidance. The EDPB guidance specifies that it includes decisions that affect someone's financial circumstances, access to services, opportunities for employment, or health. This encompasses the large majority of consequential AI decisions in commercial and government contexts.

The Right to Human Review: What Genuine Review Requires

Where Article 22's exceptions apply — allowing automated decisions that produce significant effects — the regulation requires the right to human intervention, the right to express one's point of view, and the right to contest the decision. As discussed in Chapter 15, genuine human review requires reviewers with information, expertise, time, and authority to identify and correct errors. Organizations that provide nominal human review while effectively delegating decision-making to the algorithm do not satisfy this requirement.

Enforcement attention to the quality of human review under Article 22 has been limited but is growing. The Dutch Data Protection Authority's investigation of the tax authorities' toeslagenaffaire (childcare benefits scandal) found that human review of algorithmic fraud flags was not meaningful — reviewers routinely approved the algorithm's recommendations without independent assessment — and treated this as a failure of the Article 22 requirement.

GDPR Enforcement of Article 22

Enforcement of Article 22 has been slower and less extensive than many expected when GDPR came into force. The largest fine specifically addressing Article 22 issues was the 2.75 million euro fine against the Dutch Tax Authority in the toeslagenaffaire. Several other enforcement actions have touched on Article 22 as part of broader GDPR violations without being specifically Article 22 cases.

The enforcement pattern reflects the broader challenges of GDPR enforcement: the complexity of Article 22 claims, the technical expertise required to assess whether human review is genuine, and the resource constraints of national supervisory authorities. Ireland's Data Protection Commission — which is the lead supervisory authority for most major technology companies because they have EU headquarters in Ireland — has been criticized for slow enforcement, which has significantly limited the practical impact of GDPR provisions including Article 22 on major platform companies.

Section 17.3: Academic Debate — Does Article 22 Create a Right to an Explanation?

The interpretation of GDPR Article 22 and the related provisions of Articles 13 and 14 has generated a substantial academic debate. The question is not merely legal: it goes to the fundamental question of what the right to explanation requires technically and practically.

The Goodman and Flaxman Position

Goodman and Flaxman's 2017 paper "European Union regulations on algorithmic decision-making and a 'right to explanation'" (AI Magazine) argued that GDPR creates a meaningful right to explanation that requires AI decision systems to be able to explain their outputs to affected individuals. They pointed to Recital 71's language about the right to "obtain an explanation of the decision reached" and to the "meaningful information about the logic involved" requirement of Articles 13 and 14. On this reading, GDPR requires not just notification that automated decision-making has occurred but a genuine explanation of why a particular outcome was reached.

Goodman and Flaxman's reading had significant policy influence: it shaped how the European Commission discussed GDPR, how data protection authorities initially described their enforcement priorities, and how AI developers thought about the regulatory requirements for their systems. It also focused attention on the genuine difficulty of providing meaningful explanations of complex machine learning models.

The Wachter, Mittelstadt, and Russell Counter-Argument

Wachter, Mittelstadt, and Russell's 2017 paper "Why a Right to Explanation of Automated Decision-Making Does Not Exist in European Union Law" (Harvard International Law Journal) challenged the Goodman and Flaxman reading. They argued that reading GDPR text carefully — distinguishing between Articles 13, 14, and 22, and the recitals — reveals that the regulation does not create an individual right to request an explanation of a specific automated decision ex post. What the regulation creates, they argued, is a right to information about the general logic of automated decision-making processes, provided as part of fair processing information at the time of data collection. This is a right to know, in advance, what kind of system will be making decisions — not a right to demand an explanation of a specific decision after it has been made.

The practical implications of this reading are significant. If Wachter et al. are correct, then a data subject who is denied a loan, denied a job, or denied benefits by an algorithmic system cannot demand, after the fact, an explanation of why that specific decision was made about them. They can only rely on whatever general information was provided in the organization's privacy notice.

Edwards and Veale: Counterfactuals as the Most Useful Format

Edwards and Veale's 2017 paper "Slave to the Algorithm? Why a 'Right to an Explanation' is Probably Not the Remedy You Are Looking For" accepted that GDPR's explanation provisions are limited but argued that this is less important than it might seem — because what individuals most need is not a model explanation but a counterfactual explanation: "what would have changed the outcome?"

The counterfactual explanation is practically more useful than a model explanation for several reasons. It is actionable (it tells the person what they could change). It is robust to technical opacity (it can be produced even for black-box models whose internal workings cannot be directly explained). And it is legally meaningful: knowing that "if your debt-to-income ratio had been 5 points lower, the decision would have been different" gives the person information they can use to challenge the decision (if their actual DTI is below the threshold) or to understand their situation.

Edwards and Veale's insight has been influential in the practical guidance that data protection authorities have developed. The ICO's guidance on meaningful explanation under UK GDPR emphasizes counterfactual and case-based explanation formats as particularly valuable — a notable shift from the model-explanation framing of earlier discussion.

What GDPR Has Actually Delivered

The empirical question — what has GDPR Article 22 actually delivered for individuals who have invoked it? — is sobering. Academic research and journalist investigations have found that most data subjects who invoke Article 22 rights receive responses that fall short of meaningful explanation. Organizations provide general descriptions of their modeling approaches, disclaim the provision of specific weights or scores on confidentiality grounds, and provide recourse mechanisms that often recirculate the original decision through the same algorithm. Regulatory enforcement has been insufficient to force genuine compliance, and most individuals lack the expertise to identify and contest inadequate responses.

This enforcement gap does not mean the right is worthless — it has shaped organizational practices, raised the salience of explanation in AI development, and provided a framework for advocacy and litigation. But it does mean that the right as stated and the right as delivered remain substantially different things.

Section 17.4: EU AI Act and Explanation Obligations

The EU AI Act, enacted in 2024 and phasing into full effect through 2026-2027, layers additional explanation and transparency requirements on top of GDPR for AI systems within its scope. The AI Act takes a different approach than GDPR: where GDPR focuses on data protection rights for individuals, the AI Act focuses on the risk classification of AI systems and on the obligations of providers and deployers relative to those risk classifications.

Article 13: Transparency for High-Risk AI Systems

Article 13 of the EU AI Act requires providers of high-risk AI systems to design and develop those systems with transparency so that deployers can understand them and use them appropriately. Providers must supply technical documentation and instructions for use that include: a description of the system's capabilities, limitations, and performance across relevant population groups; the data on which the system was trained and its known deficiencies; the level of accuracy and robustness expected; the human oversight measures necessary for appropriate use; and the circumstances in which the system may underperform.

This information is primarily directed at deployers — organizations that integrate high-risk AI systems into their products and services — rather than at affected individuals. The transparency requirements for deployers enable them to provide appropriate information to individuals, train users appropriately, and implement meaningful oversight. The chain of information runs from provider to deployer to affected individual, with each link in the chain responsible for conveying appropriate information downstream.

Article 86: Right to Explanation for Individual Decision-Making

Article 86 of the AI Act creates a more individual-facing right: the right to explanation for individual decisions made by high-risk AI systems. Article 86 provides that any affected person subject to a decision made with the assistance of a high-risk AI system shall have the right to obtain from the deployer a meaningful explanation of the role the AI system played in the decision-making and the main elements of the decision taken.

This provision is significant because it goes beyond what GDPR Article 22 clearly requires. It explicitly creates an individual right to explanation for specific decisions — not just general information about decision-making logic — and it applies to the full range of high-risk AI decisions as classified by the AI Act, including employment, education, essential services, law enforcement, migration and border control, and administration of justice.

Interaction With GDPR Article 22

The EU AI Act's explanation rights interact with, and in some respects extend, GDPR Article 22. Where GDPR applies to automated decisions with significant effects, the AI Act applies to the full range of decisions made by high-risk AI systems — which includes both automated decisions and decisions in which AI provides substantial assistance to human decision-makers. The AI Act's explanation right may therefore apply in cases where GDPR Article 22's "solely automated" requirement is not met — filling the gap created by nominal human involvement.

Regulatory guidance on the interaction between GDPR and AI Act explanation obligations is still developing. The likely outcome is a layered framework in which GDPR applies to automated decisions as it always has, and the AI Act adds explanation obligations for high-risk AI systems that apply regardless of the degree of automation.

Prohibited AI: Opacity as a Prohibited Feature

The EU AI Act's list of prohibited AI systems includes, notably, AI systems that deploy subliminal techniques beyond a person's consciousness to distort their behavior in ways that cause or are likely to cause harm. This prohibition treats certain forms of opacity as inherently prohibited: AI systems whose function depends on users not understanding how they work — not merely systems that happen to be opaque, but systems designed to exploit human cognitive limits — are categorically banned.

This is an important conceptual development: the AI Act recognizes that opacity is not merely a transparency failure but, in some applications, an ethically objectionable design feature. The right to explanation has a correlative: the prohibition of AI systems designed to exploit users' inability to understand them.

Section 17.5: US Law and the Explanation Gap

The United States has no federal equivalent to GDPR Article 22. American legal protection for the right to explanation in AI decision-making is partial, sector-specific, and substantially weaker than the EU framework. This gap is a significant feature of the transatlantic AI governance landscape.

ECOA Adverse Action Notices: The Closest Analog

The Equal Credit Opportunity Act's adverse action notice requirements — discussed in detail in Chapter 15 — are the closest US analog to a right to explanation in AI decision-making. When a lender takes adverse action on a credit application, the applicant has the right to specific reasons for the decision. This requirement imposes a limited explanation obligation on credit decisions, which is the most developed sector-specific explanation right in US federal law.

As noted previously, ECOA adverse action notices are inadequate for machine learning credit models: the post-hoc explanation methods used to identify "principal reasons" may not accurately represent the model's actual reasoning, and the notice format does not require disclosure of confidence levels, error rates, or the specific weights assigned to relevant factors. The CFPB has acknowledged these limitations but has not yet issued comprehensive guidance on how they should be addressed.

FCRA Dispute Rights as a Model

The Fair Credit Reporting Act (FCRA) provides consumers with rights to know what is in their credit reports, to dispute inaccurate information, and to have disputes investigated. These rights function as a limited explanation system for credit decisions: if you are denied credit based on credit report information, you can obtain your report, review it, and challenge errors. The FCRA model demonstrates that explanation rights can be operationalized in a sector-specific context — but it also illustrates the limitations: the right applies only to credit report information, not to the proprietary model that processed that information.

State-Level Initiatives

In the absence of federal action, several states have enacted or proposed AI explanation and transparency requirements.

Colorado's AI in Life Insurance Law (2021) requires insurers using AI in life insurance underwriting to establish internal oversight programs that ensure the AI does not produce results that are discriminatory or unfairly differential, and to provide explanations of adverse decisions upon request.

New York City Local Law 144 (2023) requires automated employment decision tools to undergo bias audits and requires employers to notify candidates when an AEDT was used in an employment decision — a disclosure requirement that creates limited explanation right.

Illinois's Artificial Intelligence Video Interview Act (2019) requires employers that use AI to analyze video interviews to notify applicants before using AI, obtain consent, and explain how the AI works and what characteristics it evaluates.

California's CPRA — amendments to the California Consumer Privacy Act — includes rights to know about automated decision-making and to opt out of it in certain contexts, though implementing regulations are still being developed.

Connecticut, Virginia, Colorado, and other states have enacted comprehensive privacy laws that include provisions on automated decision-making and explanation rights, though these vary significantly in scope and specificity.

The resulting patchwork of state laws creates compliance complexity for organizations operating nationally, while leaving significant gaps in federal protection. The variation across states — some requiring explanation for employment AI, others for credit AI, others for insurance AI, with different standards and enforcement mechanisms — does not produce the comprehensive framework that advocates argue is needed.

The Algorithmic Accountability Act: Proposed Federal Legislation

The Algorithmic Accountability Act has been introduced in Congress in multiple sessions, most recently in 2022, and has not passed. The Act would require companies to conduct impact assessments of "automated decision systems" that are used in consequential decisions and to take remedial action where those assessments identify discrimination or other harms. It would also require the FTC to establish rules for automated decision system transparency.

The Act's failure to pass reflects the political dynamics of US technology regulation more broadly: strong industry opposition, the difficulty of passing comprehensive technology legislation in a divided Congress, and the dominance of deregulatory impulses in the tech policy space. Whether comprehensive federal algorithmic accountability legislation will pass in the near future is uncertain.

The Transatlantic Gap and Its Consequences

The gap between EU and US explanation rights is not merely a legal technicality; it has practical consequences for individuals. A person denied a loan, a job, or healthcare in the EU has significantly more legal leverage than a person in the same situation in the US. The EU person can invoke GDPR, seek meaningful information about the logic of the decision, request human review, and complain to a data protection authority with enforcement powers. The US person must rely on sector-specific protections that are narrower, weaker, and less consistently enforced.

This gap creates competitive dynamics: companies may design AI systems for the EU market that have genuine explanation capabilities, while deploying the same systems in the US market without those capabilities, because US law does not require them. The gap may also create market access effects as the EU increasingly requires that AI systems used in its territory meet standards that US companies must choose to adopt globally or maintain as EU-specific features.

Section 17.6: Sector-Specific Explanation Rights

Beyond the general frameworks of GDPR and US federal law, several sector-specific legal regimes create explanation obligations for AI decisions. Understanding these sector-specific frameworks is essential for practitioners working in regulated industries.

Criminal Justice: Loomis and the Opacity of Risk Assessment

In State v. Loomis (Wisconsin Supreme Court, 2016), Eric Loomis challenged his sentence on the grounds that the court had relied on a COMPAS recidivism risk assessment score without disclosing the algorithm's methodology, and that this violated his due process right to know the basis of his sentence. The Wisconsin Supreme Court upheld the sentence, finding that the score was used as one factor among many in an individualized assessment and that the algorithm's general methodology was publicly known even if its specific weights were not.

The Loomis decision has been extensively criticized. Critics argue that Loomis's inability to examine the specific algorithm used in his case — a proprietary system owned by a private company — prevented him from challenging the score's accuracy or its applicability to his particular circumstances, and that this is a fundamental due process violation. The case illustrates a structural problem: the use of private, proprietary algorithmic tools in governmental decisions (sentencing, parole, pretrial detention) that have historically required reasoned explanation.

Several states have since enacted requirements that risk assessment tools used in criminal justice be validated, that their methodology be disclosed, and that defendants have access to information about the tool used in their case. California (AB 2192, 2021) requires risk assessment tools to be validated by the state, and several counties have moved away from commercial risk assessment tools entirely.

Healthcare: 21st Century Cures Act and Patient Access

The 21st Century Cures Act (2016) included information blocking provisions that require healthcare providers and health IT vendors to provide patients with electronic access to their health records — including all of the data that clinical AI tools process. While not specifically an AI explanation requirement, this provision enables patients to access the data on which AI clinical recommendations are based and to identify errors in that data — a form of algorithmic accountability even if not a direct right to explanation.

The ONC (Office of the National Coordinator for Health Information Technology) has issued rules implementing the 21st Century Cures Act that require health IT developers to ensure their systems enable patient access to clinical data, including AI-generated results and scores. Implementation has been contested and uneven, but the direction of policy is toward greater patient access to health data and the AI outputs derived from it.

Employment: The NYC Model and Its Influence

No federal law currently requires employers to explain AI-influenced employment decisions. NYC Local Law 144 represents the most developed sub-federal model: employers must notify candidates when an AEDT was used and must publicize annual bias audit results, but are not required to provide individual explanations of why a specific candidate was rejected.

The European Works Councils Directive and national co-determination laws in Germany, the Netherlands, and other EU countries provide a different model: workers' representative bodies (works councils) have rights of information and consultation when employers introduce AI systems for personnel decisions, including rights to information about how the system works and periodic review of its operation. This collective representation approach to AI explanation is more developed in Europe than individual explanation rights.

Government Benefits: Due Process Requirements

As the Arkansas Medicaid case illustrated, government benefits recipients have due process rights to meaningful notice and a genuine opportunity to be heard when their benefits are terminated or reduced. These constitutional requirements apply to algorithmic benefit decisions and require explanation capacity that the deploying agencies have often not built.

Beyond the due process minimum, the Administrative Procedure Act requires federal agencies to provide reasoned explanations for their actions — a requirement that extends to rules and systems that rely on algorithmic outputs. Courts applying this standard to agency AI systems are developing doctrine on what "reasoned explanation" requires when the reasoning is performed by a machine learning model that the agency may not be able to fully characterize.

Financial Services: ECOA, FCRA, and Their Limits

Credit decisions governed by ECOA require adverse action notices; credit reports governed by FCRA must be accessible to consumers with dispute rights; insurance decisions in some states require explanations upon request. These sector-specific protections collectively provide more AI explanation rights in financial services than in most other domains — but they remain narrowly tailored to specific decision types and fall short of meaningful explanation for complex machine learning systems.

Section 17.7: The Technical Challenge — Can AI Systems Explain Themselves?

The discussion of explanation rights in law and ethics rests on an assumption that must be examined critically: that AI systems can, in principle, provide meaningful explanations of their decisions. The technical reality is more complicated.

The Faithfulness Problem

Post-hoc explanation methods — SHAP, LIME, attention visualization, and other techniques discussed in Chapter 14 — generate approximate explanations of a model's output for a given input. These approximations are useful, but they are approximations. Research has documented that post-hoc explanations can be unfaithful to the actual model reasoning: they identify the features that most influenced the explanation method's approximation of the model's behavior, which may not be the same features that most influenced the model's actual output.

The faithfulness problem is particularly severe for complex models — deep neural networks, ensemble models — whose internal operations are genuinely difficult to characterize. A SHAP explanation of a deep neural network's credit decision might identify income and employment history as the most important features. But the model may have actually been responding to a complex nonlinear interaction between dozens of features that the SHAP approximation cannot accurately represent. If the explanation is unfaithful to the model's actual reasoning, it is misleading — even if it is technically what the explanation method produced.

The Audience Problem

What counts as a "meaningful" explanation depends entirely on the recipient. The explanation that is meaningful to a data scientist — the actual model weights, the gradient information, the feature attribution method's technical details — is not meaningful to a patient or a loan applicant. The explanation that is meaningful to a loan applicant — a plain-language description of the main reasons for the decision — may not be meaningful to a judge reviewing the decision for due process compliance.

The audience problem means that organizations cannot build a single explanation and call it meaningful. They must build explanation capability that can be translated into formats appropriate for different audiences — which is a significant organizational and technical investment that most organizations have not made.

The Gaming Problem

If explanation requirements are known in advance, models can be designed — inadvertently or deliberately — to produce explanation-compliant outputs that are not actually representative of the model's decision logic. If a model knows (in a design sense) that it must be able to produce an explanation citing specified categories of factors, it can be designed so that those factors appear important in post-hoc analysis while the actual decision is driven by other factors that would not produce a compliant explanation.

The gaming problem is not purely hypothetical. Research on adversarial attacks on explanation methods has demonstrated that models can be designed to produce misleading explanations — explanations that appear benign while the underlying model is doing something quite different. In the regulatory context, this creates risk that organizations will design systems specifically to game explanation audits rather than to be genuinely explanable.

The Rudin Argument: Use Interpretable Models Instead

Computer scientist Cynthia Rudin has argued, in a highly influential 2019 Nature Machine Intelligence paper, that for high-stakes decisions, the appropriate response to the explainability challenge is not to deploy complex models with post-hoc explanations, but to use inherently interpretable models — decision trees, logistic regression, rule sets — that can explain themselves directly, without requiring approximation.

Rudin's argument is that there is no accuracy-interpretability tradeoff in most high-stakes decision domains: carefully designed interpretable models perform comparably to complex black-box models on most tasks, when the domain has structured data, and achieve this performance while providing genuine, faithful explanations of their decision logic. The preference for complex models is often based on habit, technical fashion, or vendor marketing rather than genuine performance advantage.

The Rudin argument is compelling and has significant policy implications. If genuine explanation requires using inherently interpretable models, and if the accuracy-interpretability tradeoff is largely a myth, then organizations that claim complex models are necessary for performance and that post-hoc explanations are adequate are making claims that deserve scrutiny. In domains where interpretable models can achieve adequate performance, the choice of a black-box model with post-hoc explanations may represent an organization's preference for opacity over accountability — which is not a technically necessary choice.

Contrastive Explanations: "Why This and Not That?"

Among explanation formats, contrastive explanations — which answer the question "Why was this outcome produced rather than an alternative?" — have emerged as particularly valuable. Contrastive explanations align with how humans naturally seek to understand decisions: not "what is the model's complete logic?" but "why did my case come out this way when I expected it to come out differently?"

Research suggests that contrastive explanations are more understandable to lay audiences, more useful for identifying and challenging errors, and more robust to the faithfulness problems of post-hoc explanation methods. A contrastive explanation of a denied loan application ("Your application would have been approved if your debt-to-income ratio had been 5 points lower, or if your credit accounts had been open for an average of three additional years") gives the applicant more actionable information than a list of feature importance scores.

The EU AI Act's implementing guidance has moved toward endorsing contrastive explanation formats as a preferred approach, reflecting the influence of the Edwards and Veale line of argument on European regulatory thinking.

Section 17.8: Beyond Individual Explanation — Systemic Transparency

The focus of most discussion of the right to explanation is on individual explanation: the right of a particular person affected by a particular AI decision to receive a meaningful explanation of that decision. But individual explanation is insufficient for genuine AI accountability. Even perfect individual explanations do not reveal systemic patterns — patterns of discrimination, error concentration, or systematic bias that only become visible in aggregate data.

The Limits of Individual Explanation

Consider a credit scoring model that systematically underestimates the creditworthiness of applicants from certain zip codes — zip codes that happen to be predominantly Black and Hispanic. Each individual applicant who is wrongly denied receives an explanation identifying income and employment history as the main reasons for the decision. The explanation may be accurate for that individual case. But no individual explanation reveals the systemic pattern: that applicants from those zip codes are systematically more likely to be denied, even controlling for income and employment, because the model has encoded a geographic proxy for race.

Systemic patterns require systemic data to detect. Individual explanation, even done well, provides no visibility into aggregate patterns. This means that individual explanation rights — however robustly implemented — are insufficient for accountability at the societal level.

Aggregate Reporting Requirements

Systemic transparency requires aggregate data on AI decisions: distributional information showing how decisions are distributed across demographic groups, error rates disaggregated by population, and longitudinal analysis of trends in AI decision-making. Some regulatory frameworks require this: the EU AI Act requires fundamental rights impact assessments for high-risk AI systems, which include assessment of impacts on different population groups. ECOA requires lenders to report aggregate data on credit decisions disaggregated by race and other protected characteristics — the HMDA (Home Mortgage Disclosure Act) dataset has been crucial for fair lending research.

More systematic aggregate reporting requirements for AI decisions in employment, housing, insurance, and government benefits would significantly improve the accountability infrastructure for AI — enabling regulators, researchers, and civil society organizations to identify systemic patterns that individual explanations cannot reveal.

Algorithmic Auditing as a Complement

Algorithmic auditing — systematic independent review of an AI system's design, training data, performance metrics, and outputs — is a necessary complement to both individual explanation and aggregate reporting. Chapter 19 examines auditing in detail; the relevant point here is that auditing provides visibility into AI decision systems that neither individual explanations nor aggregate reporting can fully provide.

Third-party audits, conducted by researchers or specialized auditing firms with access to model internals, can assess faithfulness of explanations, identify sources of bias, and evaluate whether the system's actual behavior matches its stated purpose. The EU AI Act requires conformity assessments for high-risk AI systems — a form of auditing — though the standard for those assessments and the qualifications of conformity assessment bodies are still being developed.

Model Registration: The EU AI Act's High-Risk AI Database

The EU AI Act requires providers of high-risk AI systems to register those systems in a publicly accessible EU database. The database will contain information about the system's purpose, the provider's identity, performance metrics, and conformity assessment results. This registry enables regulators, researchers, civil society, and affected individuals to identify which high-risk AI systems are deployed in the EU and to access summary information about their characteristics and performance.

Model registration is a form of systemic transparency that complements individual explanation rights. It enables accountability at the system level — visibility into what systems exist and what they do — rather than only at the decision level. The EU database represents the most developed model registration framework in the world, though its implementation is still in process.

The "Explanation For Whom" Question

Systemic transparency raises the question of who the audience for transparency is. Individual explanation is designed for the person affected by a specific decision. Aggregate reporting and auditing are designed for regulators. Model registration is designed for regulators, researchers, and the public.

Civil society organizations — advocacy groups, researchers, journalists — play a crucial role in AI accountability that existing transparency frameworks often do not adequately support. Individuals affected by AI decisions often lack the technical capacity and organizational resources to identify and contest systemic harms; civil society organizations that aggregate individual experiences and conduct systemic analysis can play this role. Meaningful systemic transparency requires access mechanisms for civil society and research institutions — access to data, model information, and audit results that enables them to perform the accountability functions that individual complaint-and-enforcement systems cannot.

Section 17.9: The Global Picture

The right to explanation has been developed most robustly in the European Union, is developing incrementally and unevenly in the United States, and is being articulated in different ways by China, Canada, and other jurisdictions.

EU: Leading Through Layered Regulation

The EU leads globally on explanation rights through GDPR Article 22, now supplemented by the EU AI Act. The regulatory framework is layered: GDPR provides individual rights focused on data processing; the AI Act provides system-level obligations on providers and deployers, with individual explanation rights added in Article 86. The Digital Services Act adds transparency requirements for recommendation systems. Together, these frameworks create the most comprehensive explanation rights regime in the world.

Implementation and enforcement remain the critical challenges. GDPR has been in force since 2018 with uneven enforcement; the AI Act is phasing into effect through 2027. Whether the regulatory framework produces genuine explanation capacity in practice depends substantially on enforcement activity by national supervisory authorities and, increasingly, on litigation brought by civil society organizations and affected individuals.

US: Lagging but Moving

The US has no comprehensive federal framework for AI explanation rights. The patchwork of sector-specific rights — ECOA in credit, FCRA in credit reporting, due process in government benefits, state-level requirements in employment and insurance — provides partial protection that falls short of the EU standard. Federal comprehensive legislation has not passed; the FTC exercises limited authority through deceptive practices enforcement; and state-level experimentation is producing a diverse and uncoordinated set of requirements.

The trajectory is toward more protection — driven by increasing public awareness of AI's consequential role in individual lives, by state-level activism, and by political pressure following high-profile AI failures — but the pace is slow and the outcome uncertain. The gap between US and EU explanation rights is likely to persist for the foreseeable future, though it may narrow as US states and, eventually, Congress adopt more comprehensive requirements.

China: Algorithm Transparency Regulations

China's approach to AI transparency is distinctive. The Provisions on the Management of Algorithmic Recommendations (2022) — enacted by the Cyberspace Administration of China — require platforms that use algorithmic recommendation systems to disclose to users that they are using algorithmic recommendations and to provide users with options to view non-personalized content and to opt out of profiling-based recommendations. This is a disclosure and opt-out requirement, not an explanation right per se.

China's AI transparency regulations are notably shaped by the Chinese government's interests: they focus on algorithmic recommendations that shape content consumption (social media, news, entertainment), and they require that algorithms not be used to "endanger national security or the public interest." The transparency requirements serve, in part, the government's interest in monitoring and regulating what information citizens consume — a context that makes them difficult to compare straightforwardly with explanation rights rooted in liberal individual autonomy.

Canada: AIDA and Proposed Requirements

Canada's proposed Artificial Intelligence and Data Act (AIDA), introduced in 2022 and revised since, would require operators of "high-impact AI systems" to assess and mitigate risks, maintain documentation of those assessments, and publish plain-language explanations of their AI systems' purposes and risk mitigation measures. AIDA does not create an individual right to explanation comparable to GDPR Article 22, but it does create public transparency requirements and establishes accountability for high-impact systems.

AIDA was under parliamentary consideration as of 2024 and had not yet been enacted. Its passage and implementation would significantly develop Canada's AI governance framework.

Global Variation: Rights Strongest Where Frameworks Are Most Developed

The global pattern is clear: explanation rights are strongest and most practically meaningful where regulatory frameworks are most developed and most robustly enforced. The EU leads because it has a comprehensive legal framework with enforcement mechanisms and a culture of rights-based data protection. The US lags because it lacks a comprehensive federal framework and relies on weaker sector-specific protections. Other jurisdictions fall along a spectrum that broadly tracks the development of their regulatory infrastructure.

This pattern has implications for organizations operating globally: they must meet the most demanding requirements of the jurisdictions in which they operate, which in most cases means meeting EU standards. Organizations that build explanation capability to EU standards — and that therefore have genuine explanation capacity in their AI systems — will be better positioned for compliance as other jurisdictions move toward similar requirements.

Section 17.10: Building Explanation Capacity

The right to explanation, wherever it exists in law and ethics, must be implemented in practice. Building explanation capacity — the organizational and technical infrastructure to provide meaningful explanations of AI decisions at scale — is a significant challenge that requires investment, design choices, and cultural change.

Making Explanation Part of AI Development

As with AI communication generally (Chapter 15), explanation capacity must be built into AI systems from the design stage, not retrofitted. This means making design choices that preserve explanation capacity: maintaining records of which data points were used in each decision; selecting models that are inherently more interpretable when adequate performance can be achieved; designing model training to preserve information that post-hoc explanation methods require; and testing explanation methods during model development, not just after deployment.

It also means establishing explanation capacity as a non-negotiable design requirement — not a feature that gets added if time permits, but a quality standard that all deployed AI systems must meet. Organizations that treat explainability as a stretch goal will consistently deprioritize it under time and cost pressure. Organizations that establish it as a minimum requirement will build the capacity to meet it.

The Explanation Interface

Explanation must reach people in forms they can use. This requires design of the interfaces through which explanations are communicated — which is as much a design and communication challenge as a technical one. Well-designed explanation interfaces make key information prominent, translate technical outputs into plain language, present uncertainty in accessible formats, and provide clear pathways for recourse.

Building good explanation interfaces requires user research with representative populations of affected individuals — not just internal testing by developers who already understand the model. Explanation interfaces should be tested for comprehension, actionability, and dignity across the range of audiences who will use them.

Staff Training: From AI Output to Human Explanation

Front-line workers — the nurses, loan officers, case managers, and customer service representatives who communicate AI-influenced decisions to affected individuals — need training to provide meaningful explanations. This training must cover: what the AI system does and does not do; how to interpret its outputs; how to explain those outputs in plain language; how to answer common questions about the AI's role in the decision; and how to escalate cases where the standard explanation is insufficient.

Training must be ongoing, not one-time, and must be updated when AI systems are modified. It must be evaluated for effectiveness — through simulations, role play, or outcome monitoring — rather than treated as a compliance checkbox.

Recourse Infrastructure: Genuine Appeal Processes

Explanation rights that are not backed by genuine recourse are hollow. Building genuine recourse infrastructure — appeal processes that involve human review with information, authority, and time to identify and correct errors — is essential to making explanation rights meaningful. Recourse processes must be accessible, timely, and genuinely corrective. They must be staffed appropriately and must be monitored for effectiveness.

The feedback from recourse processes must inform AI system improvement: successful challenges indicate model errors or communication failures; patterns in challenges indicate systemic issues that may affect others who did not challenge. Building this feedback loop — from recourse back to model development — is one of the most important organizational practices for AI accountability.

Measuring Explanation Quality

Organizations that take explanation rights seriously need metrics for evaluating whether their explanations are actually working. These metrics should include both process measures and outcome measures.

Process measures assess whether explanation systems are functioning as designed: Are explanations being generated for every covered decision? Are they being delivered to affected individuals in the required timeframe? Are appeal processes being completed within the required period? Are staff using the explanation interfaces correctly? These process measures catch system failures — cases where explanations are not being generated or delivered — but do not assess whether the explanations that are generated are meaningful.

Outcome measures assess whether explanations are achieving their purpose. Testing samples of recipients for comprehension of the explanations they received is the most direct outcome measure: Do recipients understand what the explanation said? Do they know what they can do in response? Do they feel their situation has been engaged with, not just processed? Appeal rates, challenge outcomes, and patterns in what recipients ask when they seek additional information can also serve as proxy outcome measures.

Few organizations currently measure explanation quality systematically. Those that do tend to focus on process metrics — which confirm that explanations are being generated — rather than on outcome metrics that would reveal whether the explanations actually work. Building outcome measurement into explanation systems is a significant improvement over the current state of practice.

The Ethics of Explanation: Honesty About Limitations

A final dimension of building genuine explanation capacity is honesty about what explanations can and cannot tell us. Organizations that deploy AI systems have an ethical obligation not to misrepresent the quality or completeness of the explanations they provide. An organization that provides a post-hoc SHAP explanation of a complex neural network's credit decision, while internally knowing that the SHAP explanation may not faithfully represent the model's actual reasoning, is providing an explanation it knows may be misleading — even if the explanation is technically the best it can do with available tools.

Honesty about explanation limitations requires organizations to communicate: when their explanations are approximations rather than exact descriptions; when there is genuine uncertainty about which factors most influenced a specific decision; when the explanation method has known limitations for the model type being used; and when the model itself may not be able to produce faithful explanations, and a different model design might be more appropriate. This level of transparency about explanation quality is uncomfortable — it invites questions about whether the AI system should be used at all for the purpose in question — but it is what genuine ethical practice requires.

This connects back to the Rudin argument about interpretable models. If an organization cannot honestly explain its AI system's decisions, this is a signal that it may be using the wrong type of model for the decision context. The inability to provide meaningful explanations is not just a communication problem; it is a model design problem. And it is an ethical problem, because deploying an AI system in a high-stakes context without the ability to explain its decisions is deploying it without the transparency that the people it affects deserve.

Discussion Questions

1. The philosophical foundation of the right to explanation includes autonomy, dignity, and epistemic justice. Are these three foundations in tension with each other, or are they mutually reinforcing? Does your answer affect how you would design a legal right to explanation?

2. Wachter, Mittelstadt, and Russell argue that GDPR does not actually create an individual right to explanation of a specific AI decision ex post, but only a right to general information about decision-making logic. If this reading is correct, does GDPR fail to protect the interests it was meant to protect? What amendments to GDPR would better protect those interests?

3. Cynthia Rudin argues that in high-stakes domains, organizations should use inherently interpretable models rather than complex models with post-hoc explanations. Are there cases where this argument is wrong — where the performance advantage of complex models justifies the explanation disadvantage? If so, what are the ethical conditions that must be met?

4. The US lacks a comprehensive federal right to explanation for AI decisions. Should Congress enact one? If so, what would an ideal federal right to explanation statute look like? What enforcement mechanism would make it effective?

5. The Chinese government's algorithmic recommendation transparency requirements are motivated partly by the government's interest in controlling the information citizens consume. Does this render the requirements ethically different from the EU's explanation rights, even if the specific transparency requirements are similar? Can similar transparency measures serve different and even opposed political purposes?

6. Individual explanation rights are insufficient to reveal systemic patterns of discriminatory or erroneous AI decision-making. What additional accountability mechanisms — aggregate reporting, algorithmic auditing, model registration — are needed to complement individual rights? Who should have access to systemic information, and on what terms?

7. A government agency uses a proprietary vendor AI system to determine benefits eligibility. The vendor refuses to disclose the algorithm's methodology, citing trade secrets. The agency argues that it cannot provide explanation because the vendor won't allow it. What legal and contractual tools should agencies have to address this situation? What responsibilities does the agency bear for the opacity of a vendor system it uses?

This chapter has examined the right to explanation from philosophical foundation through legal framework to technical challenge and organizational implementation. The following chapter, Chapter 18, examines the related question of accountability: who is responsible when AI decisions cause harm, and what organizational and legal structures are necessary to ensure that responsibility is meaningful and enforceable.