Chapter 35: Generative AI Ethics

The Ethics of Machines That Create

Opening Hook

In the spring of 2023, a lawyer named Steven Schwartz filed legal briefs with the United States District Court for the Southern District of New York, citing six case precedents to support his client's lawsuit against the airline Avianca. The cases had names, docket numbers, citation formats, and summaries of holdings. They did not exist. Schwartz had used ChatGPT to research the law. ChatGPT had fabricated the cases — names, parties, facts, outcomes — with the calm, authoritative precision of a seasoned paralegal.

When Avianca's attorneys could not locate the precedents, they flagged them to the court. Judge P. Kevin Castel demanded explanations. Schwartz submitted an affidavit that itself contained errors. He and his supervising partner were ultimately sanctioned $5,000 and subjected to public reprimand. The New York State Bar launched an investigation. The episode became a landmark not because it was unique — AI hallucination occurs constantly — but because it made visible, in a high-stakes professional setting, a new category of AI failure: confident confabulation at scale.

Hallucination, however, is only one of many novel ethical challenges posed by generative AI. The broader category of systems that can generate convincing text, images, video, audio, and code on demand represents a qualitative shift in what artificial intelligence can do and, consequently, what harms it can enable. These systems are not merely tools that process information — they produce new content indistinguishable, at scale, from human creation. They can write like a person, speak in a person's voice, wear a person's face, and argue a position with equal fluency whether it is true or false.

This chapter examines the ethics of generative AI across the full spectrum of its challenges: hallucination and misinformation, deepfakes and synthetic media, copyright and creative labor, embedded bias, privacy violations, manipulation and persuasion, transparency obligations, and the governance responsibilities of both the organizations that build these systems and the organizations that deploy them. It is written at a moment when the technology is evolving faster than the law, faster than professional norms, and faster than our collective understanding of the harms it may cause. Epistemic humility is therefore appropriate — but so is urgency.

Learning Objectives

By the end of this chapter, students will be able to:

1. Explain what makes generative AI ethically distinct from prior generations of AI systems, including its emergent capabilities and dual-use nature.
2. Define hallucination in the context of large language models, identify its technical causes, and analyze the professional liability risks it creates in regulated domains.
3. Describe the spectrum of harms associated with synthetic media and deepfakes, with specific attention to non-consensual intimate imagery and political misinformation.
4. Analyze the copyright controversies surrounding generative AI training data and AI-generated content, including ongoing litigation and the labor displacement question.
5. Identify documented sources of bias in generative AI systems and explain how training data and design choices embed representational harms.
6. Evaluate transparency and disclosure obligations under emerging legal frameworks, including the EU AI Act and FTC guidelines.
7. Assess the governance responsibilities of organizations that build foundation models and organizations that deploy them, including appropriate use policies, human review requirements, and risk management frameworks.
8. Apply an ethical framework to real-world organizational decisions about adopting and governing generative AI tools.

Section 1: What Makes Generative AI Ethically Distinct

Artificial intelligence has been embedded in business operations for decades. Spam filters, recommendation engines, credit scoring models, fraud detection systems — all of these are AI systems that have raised ethical concerns about fairness, accountability, and transparency. But generative AI represents something different, and understanding what is different is essential to understanding its distinctive ethical challenges.

From Discriminative to Generative

The AI systems that dominated enterprise applications through the 2010s were primarily discriminative: they classified, ranked, predicted, and identified. A discriminative model takes an input and produces a label or score. A loan applicant is creditworthy or not. An image contains a cat or it does not. An email is spam or it is not. These systems operate within clearly defined output spaces, and their errors are generally failures of classification — wrong labels applied to known categories.

Generative AI systems operate differently. Rather than choosing among existing outputs, they produce new content: text, images, audio, video, code, or combinations of these. Large language models (LLMs) like GPT-4, Claude, Gemini, and Llama generate text token by token, predicting what words or characters should follow based on patterns learned from vast training corpora. Image generation systems like Stable Diffusion, Midjourney, and DALL-E produce visual content from textual descriptions through diffusion processes. These systems are not retrieving stored answers — they are synthesizing novel outputs that have never existed before.

This distinction matters ethically for several reasons. First, the output space is effectively unlimited. A discriminative model makes bounded mistakes; a generative model can produce anything — including sophisticated fabrications, harmful content, or legally actionable material. Second, generative AI systems can produce outputs that are indistinguishable from human creation at scale. Third, they can be prompted to simulate the voice, style, or identity of specific individuals, enabling a range of impersonation harms that discriminative systems could not produce.

Emergence and Capability Surprise

Large language models exhibit what researchers call emergent capabilities — abilities that appear without explicit training as models scale in size and training data. A model trained on next-word prediction develops, at sufficient scale, the ability to perform arithmetic, write legal arguments, generate computer code, solve logic puzzles, and perform rudimentary medical diagnosis. These capabilities were not programmed; they emerged.

Emergence creates a distinctive governance problem. Developers cannot fully enumerate what a model can do before it is deployed. Red-teaming and safety evaluation are necessarily incomplete, because the space of possible prompts and outputs is infinite. Novel harmful uses may be discovered only after deployment, by adversarial users exploring capabilities that developers did not anticipate. This means the conventional product safety model — test before release, certify safety, then deploy — is inadequate for generative AI. The technology's capabilities are partially unknown at deployment time.

Scale of Deployment

ChatGPT reached one hundred million users within two months of its November 2022 launch — the fastest consumer technology adoption in history. By 2024, generative AI tools had been integrated into Microsoft Office, Google Workspace, Adobe Creative Suite, Salesforce, and thousands of enterprise software applications. Tens of millions of people were using generative AI for writing assistance, image creation, code generation, legal research, medical inquiries, and professional advice.

This scale of deployment matters ethically because it transforms the stakes of individual design choices. A bias embedded in a generative model's outputs, a tendency to hallucinate in particular domains, or an inadequate content safety filter does not affect one user — it affects everyone who uses the system. When a model is used by a hundred million people, even low-rate errors produce millions of harmful outputs.

Dual-Use Nature

Generative AI is profoundly dual-use: the same capabilities that make it valuable for legitimate purposes make it dangerous in adversarial hands. The ability to generate realistic text enables both legitimate writing assistance and sophisticated phishing attacks. The ability to generate realistic images enables both creative expression and non-consensual intimate imagery. The ability to generate realistic audio enables both voice assistants and impersonation fraud. The ability to generate computer code enables both software development and malware creation.

Dual-use is not new in technology — nuclear power, the internet, and encryption all share this property. But generative AI's dual-use potential is unusually broad, accessible to non-experts, and scalable. Creating a convincing deepfake video once required sophisticated technical skills and significant resources. Today it can be accomplished in minutes with free tools. Creating personalized phishing emails at scale once required substantial human labor. Today it can be automated with a single API call.

New Stakeholders

Generative AI introduces categories of affected parties that previous AI systems did not create. Content creators — authors, artists, musicians, voice actors, journalists — are stakeholders because generative models were trained on their work and can now produce outputs that compete with them. Training data subjects — individuals whose personal information, images, or creative works were scraped from the internet — are stakeholders because they did not consent to having their data used to train commercial AI systems. These groups have begun asserting legal and ethical claims that are reshaping the landscape of generative AI governance.

The Speed-Governance Gap

Perhaps the most significant ethical challenge of generative AI is structural: the pace of deployment has dramatically outrun the pace of governance. ChatGPT was released in November 2022. The EU AI Act, the world's most comprehensive AI governance framework, was not finalized until 2024. No federal comprehensive AI law existed in the United States as of 2025. Professional bodies — bar associations, medical licensing boards, journalism standards organizations — were still formulating guidelines as the technology transformed professional practice. This gap between deployment speed and governance capacity is not an accident; it is partly a deliberate strategy by technology companies operating under the implicit assumption that markets should shape AI use before regulators intervene. Whatever its causes, the gap has resulted in generative AI being embedded at scale in professional contexts before the frameworks to govern it responsibly existed.

Section 2: Hallucination and Misinformation

The term "hallucination" in the context of large language models refers to the generation of content that is factually incorrect, fabricated, or nonsensical, presented with the same confident tone as accurate content. The word is borrowed from psychology, where it describes perceiving things that do not exist. In AI, it has a specific technical meaning: a model's output deviates from what is grounded in factual reality or the input context.

Why Hallucination Happens

Large language models do not retrieve facts from a database. They generate text by predicting which tokens are most likely to follow a given context, based on statistical patterns learned during training. This architecture is powerful but fundamentally different from a knowledge base. A model does not "know" that a case citation is accurate; it generates what a plausible case citation would look like, based on patterns in training data. If the most statistically probable completion of a legal argument is a plausible-sounding citation, the model will produce it — whether or not it corresponds to reality.

Several factors compound the problem. Training data contains errors and outdated information that the model may reproduce. Training on human-generated text teaches models to write with the confident, authoritative tone that humans use even when uncertain — which means models do not reliably signal their own uncertainty. And current models have no reliable mechanism for fact-checking their outputs against authoritative sources during generation. Retrieval-augmented generation (RAG) approaches can ground model outputs in specific documents, reducing but not eliminating hallucination. But even RAG systems can hallucinate, particularly when asked questions that go beyond the retrieved context.

The Schwartz Case in Detail

The case that made hallucination visible to the legal profession and to the public began in 2022, when a passenger named Roberto Mata sued the airline Avianca, claiming injuries caused by a metal serving cart striking his knee during a flight. Mata's attorney, Steven Schwartz, a personal injury lawyer at Levidow, Levidow and Oberman with more than thirty years of practice, was asked to prepare legal research in support of the case.

Schwartz used ChatGPT to identify cases supporting his arguments. ChatGPT produced a list of cases with citations: Varghese v. China Southern Airlines, Shaboon v. Egyptair, Petersen v. Iran Air, Martinez v. Delta Airlines, Estate of Durden v. KLM Royal Dutch Airlines, and Zicherman v. Korean Air Lines. Schwartz incorporated these citations into the brief. He later testified that he had never before used ChatGPT for legal research and had no understanding of the possibility that it could fabricate citations.

When Avianca's attorneys could not locate the cited cases in any legal database, they alerted the court. Judge Castel ordered Schwartz to produce the case materials. Schwartz, now using ChatGPT to verify the citations, asked the chatbot whether the cases were real. ChatGPT confirmed that they were, generating fake descriptions of holdings when pressed. Schwartz submitted an affidavit claiming he had verified the cases' existence — based on the chatbot's self-confirmation.

The court was not satisfied. In a May 2023 opinion, Judge Castel found that Schwartz and his supervising partner, Peter LoDuca, had submitted "false and misleading statements to the court" and had "continued to double down" on their error. They were sanctioned $5,000, required to send copies of the opinion to each judge named in the fake citations, and subjected to review by a judicial conduct committee. The New York State Appellate Division Disciplinary Committee opened an investigation.

The Confident Wrong Problem

What makes hallucination ethically serious is not merely that it produces errors — all information tools produce errors. It is that generative AI systems produce errors with exactly the same confident, authoritative tone they use for accurate outputs. There is no equivalent of a search engine returning "no results found" or a database query returning null. The system always produces something that looks like an answer. Users who are unfamiliar with the technology's limitations, or who are under time pressure, or who are inclined to trust authoritative-seeming text, are poorly positioned to detect fabrications.

This is particularly dangerous in professional contexts — law, medicine, finance, engineering — where errors can have severe consequences and where users may be deploying AI tools precisely because they do not have deep expertise in the domain.

Medical and Legal Hallucination Risks

Medical hallucination is arguably more dangerous than legal hallucination because the potential harms are more immediate. Studies have documented cases of AI systems generating plausible-sounding but incorrect diagnostic recommendations, fabricating drug interactions, and hallucinating clinical trial results. A study published in JAMA Internal Medicine in 2023 found that while AI chatbots performed better than some benchmarks on medical questions, they also produced confident incorrect answers on a meaningful fraction of queries.

The risk is highest when patients or non-expert users rely on AI outputs for medical guidance without verification, and when AI tools are embedded in clinical workflows without adequate human oversight. The clinical domain is examined more fully in Chapter 36.

Current Mitigation Approaches and Their Limits

Developers have pursued several approaches to reducing hallucination. Retrieval-augmented generation grounds model outputs in specific documents. Constitutional AI and reinforcement learning from human feedback (RLHF) can reduce certain types of harmful outputs. Chain-of-thought prompting can improve reasoning quality. System-level interventions can instruct models to express uncertainty.

None of these approaches eliminates hallucination. RAG systems can still hallucinate when queries exceed retrieved context. RLHF can make models appear more confident, not less, if evaluators reward fluent outputs. Uncertainty prompting can be inconsistent. The technical community does not currently have a reliable solution to hallucination; it has only partial mitigations. This means that professional and organizational deployment of generative AI must include human verification procedures as a structural requirement, not a discretionary option.

Liability Implications

The Schwartz case did not fully resolve the liability question, but it clarified that existing professional responsibility frameworks apply to AI-assisted work product. Professionals remain responsible for the accuracy of their work regardless of how it was produced. The ABA's Model Rules of Professional Conduct, particularly Rule 1.1 on competence, require lawyers to understand the benefits and risks of relevant technology. Courts have not yet ruled definitively on whether AI developers bear liability when their systems produce harmful hallucinations, but this litigation is developing.

Section 3: Deepfakes and Synthetic Media

Synthetic media — images, video, audio, and text generated or manipulated by AI to depict events, people, or content that does not genuinely exist — represents one of the most rapidly evolving harms in the generative AI landscape. The term "deepfake" (a portmanteau of "deep learning" and "fake") was coined in 2017 when a Reddit user with that username began posting manipulated pornographic videos using the faces of celebrities. Within a few years, the technology had proliferated from technical communities to consumer applications, and from entertainment uses to sophisticated harm.

The Spectrum from Entertainment to Harm

Synthetic media exists on a spectrum. At one end is clearly labeled creative use: historical figures in films, de-aged actors, visual effects. Moving along the spectrum are satire and parody — political deepfake videos that exaggerate for comedic effect — which exist in complex tension with free speech protections. Further along are manipulated media presented as authentic: the "shallowfake" of slowed or spliced video, and the "deepfake" of AI-synthesized faces and voices. At the far end are two categories that have mobilized legal and regulatory responses: non-consensual intimate imagery (NCII) and political deepfakes designed to deceive voters.

Non-Consensual Intimate Imagery

Non-consensual intimate imagery — creating and distributing sexualized images of real people without their consent — is the category of deepfake harm that has affected the largest number of victims and generated the most urgent legal responses. Before generative AI, NCII primarily meant distributing authentic images that had been taken or shared without consent. Generative AI added a new dimension: entirely fabricated explicit images of real people created from photographs of their clothed appearance.

The scale of this harm is documented and significant. A 2019 report by the security company Deeptrace found that 96% of deepfake videos online were non-consensual pornography. Studies tracking deepfake NCII websites documented millions of monthly visitors and catalogs of hundreds of thousands of fabricated images. The vast majority of victims are women and girls; researchers have documented that approximately 90% of NCII targets are women.

High-profile cases have included South Korean K-pop artists, American actresses, and — in a particularly alarming January 2024 episode — explicit deepfake images of Taylor Swift that circulated on X (formerly Twitter) and were viewed hundreds of millions of times before the platform took action. The Swift episode galvanized legislative attention in the United States Congress, where members introduced multiple bills addressing AI-generated NCII.

The psychological harm to victims of NCII is severe and well-documented. Victims report anxiety, depression, trauma responses consistent with sexual assault, reputational damage, and career impacts. Many describe the harm as irreversible — even when specific images are removed from one platform, they circulate elsewhere, and the victim cannot control their spread.

Jurisdictional variation in legal responses has been substantial. As of 2024, more than twenty U.S. states had enacted laws addressing NCII, with varying definitions, coverage of AI-generated content, and civil versus criminal remedies. The federal Take It Down Act, which would criminalize non-consensual sharing of intimate images including AI-generated content, was introduced in Congress in 2024. At the federal level, prior to the Take It Down Act, victims had to rely on a patchwork of laws — copyright claims over their own images, state criminal statutes where they existed, and civil claims for defamation or intentional infliction of emotional distress.

Platform policies on NCII have improved under public pressure but remain inconsistently enforced. Meta, TikTok, and Google have policies prohibiting NCII, but detection at scale is difficult, and the speed of removal often lags the speed of spread.

Political Deepfakes

In January 2024, voters in New Hampshire received robocalls using an AI-synthesized voice imitating President Biden, telling them not to vote in the Democratic primary. The calls were traced to a political consultant. The incident demonstrated that deepfake technology had reached a quality sufficient to deceive voters at scale and at relatively low cost.

Political deepfakes — synthetic media designed to influence elections — pose a distinctive threat to democratic institutions. The concern is not only about specific fabricated videos but about what researchers call the "liar's dividend": even authentic video can be dismissed as a deepfake, eroding the evidentiary value of all video evidence. Political actors can deny authentic videos of themselves; media consumers cannot reliably distinguish authentic from synthetic.

The 2024 U.S. election cycle saw significant activity: deepfake robocalls, AI-generated political advertisements, and fabricated images of candidates circulated on social media. Regulatory responses included FCC rules requiring disclosure of AI-generated content in political advertising, and state laws requiring disclosure labels on political deepfakes.

The Verification Crisis

The broader consequence of synthetic media is a verification crisis: a degradation in the shared epistemic foundation that functioning information environments require. When anyone can generate realistic video, audio, or images of events that never occurred, the cost of verifying media authenticity increases dramatically, and the ability of audiences to trust any media they have not personally verified declines. This has potential consequences for journalism, legal proceedings, democratic discourse, and interpersonal trust.

Technical responses to the verification crisis include the Content Authenticity Initiative's C2PA (Coalition for Content Provenance and Authenticity) standard, which attaches cryptographic provenance metadata to images and video. Watermarking approaches embed imperceptible signals in synthetic content to enable detection. But these approaches face adversarial pressure — watermarks can be stripped, provenance metadata can be manipulated — and require broad adoption across hardware and software platforms to be effective.

Section 4: Copyright and Creative Labor

The training of large generative AI models has required ingesting vast quantities of text and images from the internet. The legal and ethical status of that ingestion — and of the outputs it enables — has become one of the most actively litigated questions in intellectual property law.

The Training Data Copyright Controversy

Large language models and image generation systems are trained on billions of documents scraped from the internet: books, news articles, social media posts, academic papers, creative writing, code repositories, and more. This data was not, for the most part, licensed from its creators. The developers of these models argue that training constitutes "fair use" under copyright law — a transformative process that produces something fundamentally different from any individual training example. Rightsholders argue that the use of their copyrighted work without compensation or permission violates their rights.

The legal questions are genuinely unsettled. Fair use doctrine requires a four-factor analysis considering the purpose and character of the use, the nature of the copyrighted work, the amount and substantiality of the portion used, and the effect on the market for the original. Different cases present these factors differently, and the law is not yet settled.

NYT v. OpenAI

The most prominent lawsuit is The New York Times Company v. Microsoft Corporation and OpenAI, filed in December 2023. The Times alleges that OpenAI trained its models on millions of Times articles without authorization, and that the models can reproduce near-verbatim Times content in response to prompts, directly competing with the Times's own products. The suit seeks billions of dollars in damages and raises the question of whether AI companies can build commercial products on the work of news organizations without compensation.

The case is significant beyond its immediate parties. The Times has documented specific instances in which ChatGPT reproduced lengthy Times articles nearly verbatim — suggesting not just that training was conducted on Times content, but that the model has memorized and can reproduce protected expression. This evidence may prove particularly relevant to the fair use analysis, as courts look at whether the use supplants the market for the original.

Artist Class Actions

Parallel to the Times litigation, a class action lawsuit brought by artists Sarah Andersen, Kelly McKernan, and Karla Ortiz challenged Stability AI, Midjourney, and DeviantArt for training image generation models on artists' work without consent or compensation. The artists alleged that the models had learned to replicate their distinctive styles, enabling users to generate images "in the style of" specific living artists — potentially competing with the artists themselves.

The "style" question is legally complex. Copyright protects specific expression, not ideas or styles. The general artistic style of an artist is not copyrightable. But the ability to generate images that closely imitate an artist's distinctive approach may constitute a commercial harm even if it does not constitute literal copyright infringement, particularly if the generated images compete in the same market.

AI-Generated Content and Copyright

The copyright status of AI-generated content is a separate question. In Thaler v. Perlmutter, a federal district court ruled in 2023 that an AI-generated image — created autonomously by Dr. Stephen Thaler's "Creativity Machine" without human authorship — could not receive copyright protection. The court held that copyright requires human authorship, and that content created entirely by an AI system falls outside copyright protection.

The implications of Thaler are significant: content generated by AI without substantial human creative input may be in the public domain, unable to receive copyright protection. This has implications for businesses investing in AI-generated creative content: they may be unable to protect that content from reproduction by competitors. It also raises questions about AI-assisted content in which a human provides prompts or selects among outputs — questions that the Copyright Office and courts are still working through.

The Labor Displacement Question

The legal questions are inseparable from an economic reality: generative AI has substantially reduced the demand for certain categories of human creative labor. Stock image markets have contracted as image generation tools have enabled users to create custom images on demand. Clients who once hired copywriters for routine content work now use AI tools. Voice actors have seen the market for licensed voice recordings disrupted by AI voice synthesis.

The entertainment industry confronted this reality most publicly during the 2023 Writers Guild of America and Screen Actors Guild strikes, which included AI as a central issue alongside compensation. The WGA obtained contract language establishing minimum protections around AI use: studios could not use AI to generate scripts or use writers' work to train AI without consent. SAG-AFTRA similarly negotiated protections for performers' digital likenesses, establishing consent and compensation requirements for AI-generated performances that imitate specific actors.

Section 5: Bias in Generative AI

The bias problems that affect discriminative AI systems — discussed in earlier chapters in the context of lending, hiring, and criminal justice — also affect generative AI, but with distinctive characteristics. Generative AI bias is not primarily about differential error rates across demographic groups in a classification task. It is about stereotype amplification, representational harm, and the normative assumptions embedded in what models produce.

Stereotype Amplification in Text

Large language models trained on internet text absorb the statistical patterns of that text, including its stereotypes, biases, and discriminatory associations. Studies have documented that LLMs associate certain professions with specific genders, associate certain names with criminality, produce more negative sentiment in descriptions of some racial groups than others, and default to male pronouns for professionals and female pronouns for caregivers. These are not neutral statistical patterns — they reflect and potentially reinforce existing social inequalities.

The mechanism of stereotype amplification is important: models do not simply reproduce stereotypes from their training data. They can amplify them, producing outputs that are more stereotyped than the average of the training data, because the most statistically probable completion of a stereotyped prompt is the most stereotyped version of the stereotype.

Racial and Gender Bias in Image Generation

Image generation systems exhibit particularly visible bias in how they represent human beings. Studies of systems including DALL-E, Stable Diffusion, and Midjourney have documented systematic patterns: when prompted to generate images of "a doctor" or "a CEO," systems predominantly generate images of white men; when prompted to generate images of "a criminal" or "a homeless person," systems generate images of darker-skinned individuals at higher rates; when prompted to generate images without explicit demographic specifications, systems default to particular racial presentations.

These patterns matter because image generation systems are increasingly used for visual content in marketing, journalism, education, and business communications. Systematically biased outputs can perpetuate the very stereotypes they reflect.

The Gemini Controversy

In February 2024, Google's Gemini image generation tool generated significant controversy when users discovered that it was producing racially diverse images in historical contexts where diversity was historically inaccurate — including diverse groups of Nazi German soldiers and diverse groups of American Founding Fathers. The controversy revealed a tension in bias correction strategies: attempts to increase diversity in image outputs can produce historically inaccurate results if applied without contextual sensitivity.

Google suspended Gemini's ability to generate images of people while it worked on corrections. The episode illustrated that bias mitigation in generative AI is not simply a matter of adding more diversity to outputs — it requires contextual judgment that generative models do not consistently exercise.

Representation as a Design Choice

The Gemini episode, and the broader body of research on representational bias, makes clear that how generative AI systems represent people is a design choice with ethical implications. Developers must decide how to weight competing values: historical accuracy, contemporary representation norms, mitigation of harmful stereotypes, and contextual appropriateness. There is no neutral default; every representation choice reflects a set of values. The ethical question is not whether to make these choices but how to make them transparently, accountably, and in ways that minimize harm.

Section 6: Privacy and Data in Generative AI

Generative AI creates several distinct categories of privacy risk that differ from those created by traditional data-intensive AI systems.

Training Data Privacy

The vast datasets used to train generative AI models typically contain substantial personal information: names, addresses, social media posts, photographs, medical records in some cases, email content, financial information, and more. This information was collected for specific purposes — social networking, news publishing, academic research — not for training commercial AI systems. The use of personal information to train AI models without the knowledge or consent of the individuals whose information is included raises significant questions under data protection frameworks.

GDPR's application to AI training data is an active area of regulatory development. The Italian data protection authority temporarily suspended ChatGPT in 2023 over concerns about the lawfulness of personal data processing in training. Other European data protection authorities have launched investigations. The core question — whether scraping publicly available personal information to train AI models is lawful under GDPR — has not been definitively resolved.

Generating Content About Real People

Unlike traditional databases, generative AI can produce new content about specific individuals: descriptions of events they did not participate in, statements they did not make, images of situations that never occurred. This creates privacy harms that did not exist with prior technologies. A database might reveal accurate private information; a generative model can create plausible-seeming false private information.

Memorization and Reconstruction Attacks

Research has documented that large language models memorize specific content from training data and can be induced to reproduce it. Studies have demonstrated that models can reproduce verbatim portions of copyrighted books, personal email addresses, social security numbers, and other specific sensitive content from training data when appropriately prompted. This "training data extraction" represents a privacy risk: data that was collected for one purpose and processed in training can be extracted in a form that reveals information about individuals.

The Right to Erasure Problem

GDPR's right to erasure — the "right to be forgotten" — creates a technically challenging obligation for AI systems. Traditional databases can delete a record. LLMs cannot easily "unlearn" specific training data without retraining the entire model. If an individual requests deletion of their personal data under GDPR, and that data was used to train a model, how does the organization comply? This is an unsolved technical and legal problem. Some approaches to machine unlearning are being developed, but none yet provides the clean erasure that data protection law contemplates.

Section 7: Manipulation and Persuasion

Among the most consequential ethical concerns about generative AI is its potential to enable manipulation and persuasion at scale — a capability that threatens democratic discourse, consumer protection, and personal autonomy.

AI-Powered Persuasion at Scale

Previous persuasion technologies — advertising, political campaigning, propaganda — required human labor to produce persuasive content. Generative AI dramatically reduces this labor requirement. A single operator can use AI tools to generate thousands of personalized persuasive messages, tailored to individual recipients based on their documented preferences, beliefs, and vulnerabilities. A political campaign can generate millions of personalized voter outreach messages. A foreign disinformation operation can produce a daily output of persuasive articles that would have required a small newsroom to produce manually.

Personalized Manipulation

Personalization — adapting messages to individual recipients — is a powerful tool of persuasion. Research on persuasion has documented that messages tailored to individuals' psychological profiles, values, and concerns are substantially more effective than generic messages. Generative AI, combined with the large behavioral datasets accumulated by social media platforms and data brokers, enables personalized persuasion at a scale previously impossible.

The Federal Trade Commission has expressed concern about AI-enabled personalized manipulation, particularly in the context of consumer marketing. FTC guidance has warned that using AI to exploit consumer vulnerabilities — including emotional states, psychological profiles, or addiction patterns — may violate Section 5 of the FTC Act's prohibition on unfair or deceptive trade practices.

Voter Targeting and Electoral Integrity

The combination of AI-powered persuasion, micro-targeting capabilities, and deepfake media creates a toolkit for electoral manipulation that is unprecedented in scope. Researchers have documented AI-generated political disinformation campaigns, fake social media accounts operated at scale, and AI-generated political advertising in multiple electoral contexts worldwide.

Phishing and Social Engineering

At the individual level, generative AI has substantially enhanced the sophistication of phishing attacks and social engineering. Traditional phishing emails were often identifiable by poor grammar, generic salutations, and implausible scenarios. AI-generated phishing can be grammatically perfect, personalized to the recipient's known interests and professional context, and contextually plausible. Security researchers have documented AI-generated spear phishing attacks that were substantially more effective than traditional approaches.

Section 8: Transparency and Disclosure

As generative AI systems become embedded in the production of text, images, and audio across professional and public contexts, questions about when and how AI involvement must be disclosed have become urgent.

The FTC's Approach

The Federal Trade Commission has taken an active interest in AI-generated content as a consumer protection issue. FTC guidance has emphasized that using AI to generate content — product reviews, testimonials, endorsements, advertising — without disclosure may constitute deceptive trade practices if it misleads consumers about the nature and authenticity of the content. In particular, the FTC has focused on AI-generated fake reviews, which it has targeted through both guidance and enforcement actions.

The EU AI Act's Transparency Requirements

The EU AI Act, which entered into force in 2024, establishes specific transparency obligations for generative AI. Systems that interact with humans must disclose that they are AI. AI-generated content — specifically text, audio, images, and video — must be labeled as AI-generated in ways that are "technically feasible and effective." Providers of general-purpose AI models, including foundation model developers, must publish summaries of training data.

The C2PA Provenance Standard

The Content Authenticity Initiative, a project of Adobe, Microsoft, and other technology companies, has developed the C2PA (Coalition for Content Provenance and Authenticity) open standard for attaching cryptographic provenance metadata to media. When a camera takes a photograph, it can embed a signed record of that photograph's origin. When an AI system generates an image, it can embed a signed record of its AI generation. This metadata chain allows downstream consumers to verify the origin of media.

C2PA adoption is growing: major camera manufacturers have announced C2PA support, and several AI image generation systems embed C2PA metadata. But the standard faces challenges: metadata can be stripped by image compression or platform processing, and bad actors can simply generate content without metadata. The standard is a useful tool but not a complete solution.

Watermarking Approaches

AI watermarking — embedding imperceptible signals in AI-generated content — is another approach to transparency. These signals can enable detection of AI-generated content even when metadata has been stripped. Google's SynthID system embeds imperceptible watermarks in AI-generated images, audio, and text. OpenAI has researched text watermarking approaches. However, watermarks can potentially be removed through adversarial techniques, and the robustness of current watermarking approaches is an active research question.

Section 9: Foundation Model Responsibility

The development of large foundation models — models trained at enormous scale that are then deployed across thousands of applications — creates a distinctive governance structure in which responsibility for downstream harms is distributed across a chain of actors.

The Responsibility Chain

A foundation model like GPT-4 or Llama is developed by one organization (OpenAI, Meta). It is accessed by application developers through an API or downloaded for local deployment. Application developers build products — customer service chatbots, writing assistants, code generators — using the foundation model. End users interact with these products. When harm occurs — a hallucination, a harmful output, a privacy violation — who is responsible? The foundation model developer, the application developer, or the deploying organization?

The Acceptable Use Policy Model

Foundation model developers primarily govern downstream use through terms of service and acceptable use policies. OpenAI's usage policies prohibit specific categories of use: weapons of mass destruction, CSAM, election interference, and others. Anthropic's usage policies similarly prohibit specific harmful applications. Violation of these policies can result in termination of API access.

This model places significant responsibility on developers but relies on contractual enforcement rather than technical prevention. Prohibited uses can occur if developers or users violate policies, and enforcement at scale is challenging. The policies also leave substantial gray areas — dual-use applications that can serve both legitimate and harmful purposes.

Safety Approaches

The major frontier AI developers have invested in safety research and pre-deployment evaluation, including adversarial testing (red-teaming), constitutional AI approaches that train models to follow ethical principles, and reinforcement learning from human feedback to align model behavior with human values. These approaches reduce certain categories of harm but are not foolproof.

Open vs. Closed Access

A significant governance debate concerns whether models should be released openly (weights published, anyone can download and run them) or through controlled API access. Open release democratizes access, enables research and customization, and prevents single-company control of critical infrastructure. It also enables use cases that the original developer could not prevent: removing safety filters, fine-tuning for harmful applications, and deploying on private infrastructure without oversight.

Meta's release of the Llama model family has been the most prominent example of open model release. Researchers and safety advocates have disagreed about whether open release increases or decreases overall harm. This debate remains unresolved, and different organizations have reached different conclusions based on different assessments of the risk-benefit tradeoff.

Section 10: Organizational Governance of Generative AI

For business professionals, the ethical questions about generative AI are not only abstract policy questions — they are practical governance challenges that every organization deploying these tools must address.

Enterprise Use Policies

The first governance priority is establishing clear organizational policies about how generative AI tools may and may not be used. These policies must address: which AI tools are approved for organizational use; what categories of information may be entered into AI systems (given that inputs may be used for model training); what outputs require human review before use; and what uses are prohibited.

Prohibited uses typically include entering confidential client information, proprietary business data, or employee personal information into third-party AI systems without appropriate data processing agreements. They also include generating false content, impersonating individuals, bypassing legal or regulatory requirements, and engaging in discriminatory applications.

The Shadow AI Problem

A persistent challenge is "shadow AI" — employee use of AI tools that are not approved or monitored by the organization. Just as the early smartphone era produced "shadow IT," the generative AI era has produced widespread unsanctioned use of AI tools in workplaces. Employees use personal accounts with commercial AI tools to accelerate work, entering confidential information into systems that may not meet organizational security or data protection requirements. This creates legal, security, and ethical risks that are difficult to manage when the organization lacks visibility into AI use.

Vendor Evaluation

Procuring generative AI tools from third-party vendors requires a governance framework for vendor evaluation. This framework should assess: the vendor's data processing terms (whether inputs are used for training, data retention policies, security certifications); the model's performance characteristics relevant to the intended use case (accuracy, hallucination rate); the model's safety properties; the vendor's transparency about model development and known limitations; and the regulatory compliance status of the vendor's products in relevant jurisdictions.

Appropriate Use Cases and Prohibited Uses

Not every use case is appropriate for generative AI. Organizations should identify use cases where the technology's limitations are acceptable — where errors can be caught by human review, where stakes are low enough to tolerate imperfect outputs, and where the benefits are clear. They should identify prohibited uses — high-stakes decisions in legal, medical, or financial contexts where AI hallucination could cause serious harm — and require human expertise for these decisions.

Human Review Requirements

A consistent principle across governance frameworks for generative AI is the requirement for human review of AI outputs in consequential contexts. The Schwartz case illustrates the failure mode: a professional relying on AI output without independent verification. Governance frameworks should establish, for different categories of AI-assisted work, what human review is required, who is qualified to conduct it, and how it should be documented.

The Generative AI Risk Management Framework

NIST's AI Risk Management Framework (AI RMF), published in January 2023, provides a structure for organizational AI governance that can be applied to generative AI. The framework organizes AI risk management around four functions: Govern (establishing organizational roles, policies, and culture for AI risk management), Map (identifying and categorizing AI risks), Measure (analyzing and assessing risks), and Manage (prioritizing and treating risks through policies and procedures).

Organizations adopting generative AI tools should map specific use cases to this framework, identifying the categories of risk each use case presents, establishing measurement approaches to monitor those risks, and implementing management controls appropriate to the risk level.

Ethics Washing in Generative AI

A recurring concern in the generative AI space is "ethics washing" — the use of ethical language, governance structures, and safety commitments to signal responsible development without substantive changes in practice. Organizations developing and deploying generative AI have strong commercial incentives to appear responsible, and the reputational cost of being seen as irresponsible is significant. These incentives can produce responsible-sounding policies that do not constrain harmful practices in meaningful ways.

Indicators of ethics washing include: ethics boards that are advisory rather than decision-making; safety teams that are overruled by commercial teams on deployment decisions; responsible AI principles stated at a level of generality that commits the organization to nothing specific; and transparency reports that disclose information about lower-risk model properties while obscuring information about higher-risk ones. Business professionals evaluating AI vendors or partners should be attentive to the gap between stated principles and operational practice.

Recurring Themes in This Chapter

Power and Accountability: Generative AI concentrates enormous creative and persuasive power in organizations that develop foundation models, and distributes that power — along with accountability gaps — across long chains of deployers and users. The architecture of foundation models makes it structurally difficult to hold any single actor fully responsible for harms.

Innovation vs. Harm: The genuine productivity benefits of generative AI — in writing, code generation, accessibility, and creative assistance — exist in real tension with documented harms. The ethical challenge is not to choose innovation over harm or harm prevention over innovation, but to develop governance structures that preserve genuine benefits while preventing or mitigating serious harms.

Ethics Washing: The generative AI industry has produced significant ethics theater — stated commitments to safety, published principles, prominent research on harms — alongside deployment decisions that prioritize scale and revenue over demonstrated safety. Scrutinizing the gap between statement and practice is essential.

Diversity and Inclusion: Representational bias in generative AI systems both reflects and risks reinforcing existing social inequalities. The creative labor displacement caused by generative AI has differential impacts across communities. The governance of generative AI has been dominated by a small number of highly capitalized technology companies, with limited input from the communities most affected by its harms.

Global Variation: Regulatory responses to generative AI vary dramatically across jurisdictions. The EU AI Act establishes binding obligations; the United States has taken a more voluntary approach at the federal level; China has established registration requirements for generative AI products. Organizations operating across jurisdictions must navigate this regulatory variation while maintaining coherent governance standards.

Conclusion

Generative AI is not a future technology. It is a present technology, embedded at scale in professional practice, consumer products, and public information environments. Its ethical challenges — hallucination, deepfakes, copyright violation, bias, privacy violation, manipulation, and opacity — are not hypothetical risks. They are documented harms occurring in real professional contexts, affecting real people.

The governance response to these harms is developing, but slowly, and in fragmented ways across jurisdictions and sectors. Professional bodies are establishing standards. Legislatures are enacting laws. Courts are resolving disputes. Regulators are issuing guidance. But the pace of governance has lagged the pace of deployment, and significant harms have occurred — and continue to occur — in that gap.

For business professionals, the practical implication is clear: the deployment of generative AI requires active governance, not passive adoption. Organizations that integrate generative AI without clear use policies, vendor evaluation standards, human review requirements, and ongoing monitoring are taking on legal, reputational, and ethical risks. The technology's genuine benefits are worth pursuing — but pursuing them responsibly requires confronting, rather than minimizing, the genuine harms it can cause.

Key Terms

Hallucination: The generation by an AI system of content that is factually incorrect, fabricated, or nonsensical, presented with the same confident tone as accurate content.

Deepfake: Synthetic media — images, video, or audio — in which a person's likeness or voice has been manipulated or entirely generated by AI.

Non-Consensual Intimate Imagery (NCII): The creation and distribution of sexualized images of real people without their consent, including AI-generated images.

Foundation Model: A large AI model trained at scale on broad data and deployed across many downstream applications.

Training Data Extraction: The ability to induce a trained AI model to reproduce verbatim content from its training data.

Shadow AI: Employee use of AI tools that are not approved or monitored by the organization.

C2PA: The Coalition for Content Provenance and Authenticity standard for attaching cryptographic provenance metadata to media.

Retrieval-Augmented Generation (RAG): An approach that grounds language model outputs in specific retrieved documents to reduce hallucination.

Ethics Washing: The use of ethical language and governance structures to signal responsible development without substantive changes in practice.

Right to Erasure: A data protection right under GDPR enabling individuals to request deletion of their personal data.