Case Study 1: The Chatbot That Passed the Bar Exam

What Happened

In early 2023, OpenAI announced that GPT-4 had scored in the 90th percentile on the Uniform Bar Examination — the test that aspiring lawyers must pass to practice law in the United States. The news sent shockwaves through the legal profession and the general public. Headlines proclaimed that AI could now "pass the bar exam," with the implication that AI was approaching — or had already reached — lawyer-level intelligence.

The claim was technically accurate. GPT-4 did score at that level on the multiple-choice and essay portions of a simulated bar exam. Its predecessor, GPT-3.5, had scored in the bottom 10th percentile just months earlier. The improvement was staggering, and the result was real.

But what did it actually mean?

The Performance

Let's look at what GPT-4 actually did. The Uniform Bar Exam has three components:

  1. The Multistate Bar Examination (MBE): 200 multiple-choice questions covering seven areas of law. GPT-4 performed strongly here — pattern matching across legal concepts, identifying the "best answer" among options.

  2. The Multistate Essay Examination (MEE): Six essay questions requiring analysis of legal scenarios. GPT-4 generated well-structured essays that identified relevant legal issues and applied appropriate rules.

  3. The Multistate Performance Test (MPT): Two tasks requiring the test-taker to use a provided set of documents (case files, statutes, memos) to complete a practical legal task. This is designed to test skills that go beyond memorization.

GPT-4's performance was impressive across all three components. But several important caveats tend to get lost in the headlines.

What the Headlines Missed

The bar exam tests a specific, bounded kind of knowledge. It asks questions with determinable correct answers within established legal frameworks. It doesn't test the ability to meet with a frightened client, assess their credibility, develop case strategy based on incomplete information, negotiate with opposing counsel, or make ethical judgments under pressure. These are the activities that constitute most of a lawyer's actual work.

The model doesn't "know" law — it predicts legal-sounding text. GPT-4 was trained on enormous quantities of legal text: case law, legal textbooks, bar prep materials, law review articles. When presented with a bar exam question, it generates the most statistically probable response based on those patterns. This produces correct answers much of the time, but the process is fundamentally different from a law student who has spent three years learning to "think like a lawyer" — understanding the purposes behind legal rules, the policy tensions that inform judicial decisions, and the ethical obligations that constrain legal practice.

Benchmark performance can be misleading. When researchers have probed more deeply — asking LLMs to reason through novel legal scenarios that don't closely resemble their training data, or to identify subtle logical errors in legal arguments — performance drops significantly. The model excels on questions that resemble its training data and struggles on genuinely novel problems.

The model cannot be held accountable. A human lawyer who gives bad legal advice can be sued for malpractice, disciplined by the bar, or even disbarred. These accountability structures are fundamental to the legal profession's social contract: you trust your lawyer in part because they face real consequences for incompetence. An LLM faces no such consequences. It has no license to revoke, no career to lose, no ethical obligations.

The Deeper Question

The bar exam result raises a question that goes far beyond law: What do standardized tests actually measure, and what do we lose when we confuse test performance with competence?

Standardized tests, by design, assess a narrow band of capability. They're useful shortcuts — a way to verify that a person has achieved a baseline level of knowledge. But they've never been the whole picture. No one would say that passing the bar exam is sufficient to be a good lawyer. It's necessary, but it's a starting point.

When an LLM "passes" such a test, it demonstrates that next-token prediction on legal training data can produce outputs that match the test's expected answers. It does not demonstrate understanding, judgment, empathy, ethical reasoning, or any of the other capacities that make a lawyer a lawyer.

This same pattern has repeated across domains. LLMs have passed medical licensing exams, MBA exams, AP tests, and graduate-level science assessments. In each case, the result is genuinely impressive — and in each case, the gap between test performance and professional competence remains vast.

The Real-World Impact

Despite these caveats, LLMs are already transforming legal practice. Law firms use them to:

  • Draft initial versions of contracts, briefs, and memos
  • Summarize large volumes of case law during legal research
  • Identify relevant precedents across thousands of documents
  • Generate first drafts of client communications

These are genuinely valuable applications. They save time, reduce costs, and in some cases improve access to legal services for people who couldn't previously afford them. The technology is useful precisely when it's used as a tool rather than a replacement — when a human lawyer reviews, verifies, and takes responsibility for the output.

The danger arises when the headline — "AI passes the bar!" — leads people to treat LLM output as equivalent to legal advice. Self-represented litigants using chatbots to draft court filings have already submitted documents containing hallucinated case citations. In one well-publicized 2023 incident, a lawyer submitted a brief to federal court containing six fake cases generated by ChatGPT. The lawyer was sanctioned by the judge.

Discussion Questions

  1. If an LLM scores higher than most human test-takers on the bar exam, does that mean it "understands" law? What additional evidence would you need to make that claim?

  2. Should LLM performance on professional exams change how we think about those exams? Is the problem with the AI or with our exams?

  3. A startup offers a service where people can get "AI legal advice" at a fraction of the cost of a human lawyer. The AI passes the bar exam at the 90th percentile. Who benefits from this service? Who might be harmed? What safeguards would you want in place?

  4. Connect this case to the chapter's threshold concept: "LLMs predict the next word — they don't understand meaning." How does the bar exam result illustrate both the power and the limitation of next-token prediction?

  5. How does this case relate to Priya's experience with the fake citation? What common thread connects the two stories?