Case Study 8.1: The AI Lawyer That Cited Fake Cases

LLM Hallucinations in Professional Settings

The Setup

In June 2023, a federal judge in the Southern District of New York imposed sanctions on two lawyers at the law firm Levidow, Levidow & Oberman for submitting a legal brief that cited six court cases that did not exist. The cases had plausible-sounding names, correct citation formats, and were attributed to real courts. They included fabricated quotes from fabricated judicial opinions. None of them were real.

The lawyer who prepared the brief, Steven Schwartz, had used ChatGPT to conduct legal research. He asked the chatbot to find relevant case law supporting his client's argument in a personal injury lawsuit against Avianca Airlines. ChatGPT obliged, producing multiple citations with impressive specificity — case names, docket numbers, dates, and passages from opinions.

Schwartz later testified that he had asked ChatGPT to confirm the cases were real. ChatGPT confirmed they were. He asked it again to make sure. It confirmed again. He did not verify the citations through a legal database like Westlaw or LexisNexis. He submitted the brief to the court.

What Went Wrong

This case is a near-perfect illustration of several concepts from Chapter 8:

Hallucination at its most dangerous. ChatGPT didn't just make up case names. It generated complete fabrications in the exact format that a real case would take: proper citation style, plausible court names, realistic holdings, and even fabricated quotes from nonexistent judicial opinions. The hallucinations were so well-structured that they passed an initial review by the lawyer, his colleagues, and even his partner — all of whom assumed the citations were real because they looked real.

The self-verification trap. When Schwartz asked ChatGPT to verify its own citations, he fell into a trap that anyone might fall into: asking the same system that generated an output to verify that output. ChatGPT has no mechanism for checking its outputs against a legal database. When asked "Is this case real?", it's essentially predicting what token comes next after that question in the context of a conversation about that case — and the most probable next token is some form of "yes." The model was confirming its own hallucinations through the same mechanism that generated them.

Automation bias in a professional context. Schwartz was an experienced attorney with over 30 years of practice. He knew how to verify cases — he had done it thousands of times using traditional legal databases. But the ease and apparent authority of the AI tool changed his behavior. He described ChatGPT as being like a "super search engine," a characterization that fundamentally misunderstands what the technology does. A search engine retrieves existing documents; an LLM generates new text based on patterns. The distinction matters enormously.

The Court's Response

Judge P. Kevin Castel's opinion sanctioning the lawyers was notable for its clarity about the nature of the failure:

"Technological advances are commonplace and have long been adapted to the practice of law. There is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings."

The judge emphasized that the problem wasn't using AI — it was failing to verify AI output. The attorneys were sanctioned not because they used ChatGPT, but because they submitted fabricated citations to a federal court without checking whether those citations were real. In legal terms, they violated their duty of candor to the tribunal.

The sanction was $5,000 — modest in financial terms but devastating in professional terms. The case made international headlines and became the most widely cited example of AI hallucination causing real-world harm.

The Ripple Effects

The Avianca case triggered a cascade of responses across the legal profession:

Judicial responses. Within months, federal judges across the United States began issuing standing orders about AI use in their courtrooms. Some required lawyers to disclose any use of AI in preparing filings. Others required lawyers to certify that all citations had been verified by a human against an authoritative legal database. A few banned AI-generated filings entirely.

Bar association guidance. State bar associations began issuing ethics opinions on AI use. Most took a measured position: AI tools are acceptable, but lawyers remain responsible for the accuracy of their work. Using AI does not reduce the duty to verify.

Legal technology adaptation. Legal technology companies began developing AI tools specifically designed to ground their outputs in real legal databases — combining LLM capabilities with retrieval-augmented generation to reduce (though not eliminate) the hallucination risk. These tools were explicitly marketed as alternatives to general-purpose chatbots for legal research.

The broader precedent. The case became a reference point far beyond law. In medicine, journalism, academia, and other professions where factual accuracy is critical, the Avianca case served as a warning: AI tools can produce outputs that are structurally correct and substantively wrong, and professional responsibility for verification cannot be delegated to the machine.

What This Case Teaches About Hallucination

The Avianca case is worth studying because it illustrates several principles that apply far beyond law:

  1. Format is not substance. A well-formatted output — correct citation style, proper structure, appropriate terminology — does not indicate factual accuracy. LLMs are extraordinarily good at mimicking the form of expert output. They are not reliably good at producing the substance.

  2. Self-verification fails. Asking an LLM to verify its own outputs is like asking a parrot to confirm that what it just said is true. The parrot doesn't know. The model doesn't know. Verification must come from an independent source.

  3. Expertise doesn't immunize against automation bias. Schwartz was a 30-year veteran lawyer. His expertise in law did not prevent him from being fooled by a technology he didn't fully understand. Domain expertise is necessary for verification, but it's not sufficient if you don't know the limitations of the tool you're using.

  4. Professional responsibility survives automation. The court's ruling made clear that using an AI tool does not transfer responsibility from the professional to the machine. The attorney is still the one who signs the filing. The doctor is still the one who signs the diagnosis. The journalist is still the one whose byline appears on the story. Responsibility stays with the human.

Discussion Questions

  1. The verification failure. Schwartz asked ChatGPT to verify its own citations and believed the confirmation. Design a verification protocol for lawyers using AI research tools that would prevent this type of error. How many steps should it have? What specific actions should it include?

  2. The expertise gap. Schwartz described ChatGPT as a "super search engine." Why is this characterization wrong? How does this misunderstanding connect to the capability-vs.-understanding theme from the broader textbook? What is the minimum level of AI literacy a professional should have before using AI tools in high-stakes work?

  3. Proportionate response. Some judges banned AI-generated filings entirely. Others required disclosure and verification. Which approach is more appropriate? What are the risks of overreaction? What are the risks of underreaction?

  4. Extending the lesson. This case involved law, but the same dynamics apply to medicine, journalism, academia, and engineering. Choose one of these fields and describe a plausible scenario where an AI hallucination could cause comparable harm. What safeguards specific to that field would you recommend?

  5. Connecting to the chapter. This case study illustrates hallucination, automation bias, and the failure of self-verification. How does the VERIFY framework from Section 8.6 apply? Walk through each letter of VERIFY as it relates to Schwartz's situation.

Key Takeaway

The Avianca case demonstrates that AI hallucinations are not a theoretical risk — they are an immediate, practical danger in any professional context where factual accuracy matters. The case also demonstrates that the solution is not to avoid AI tools, but to use them with a clear understanding of their limitations and a commitment to independent verification. The machine produces the draft. The human bears the responsibility.