Chapter 5: Large Language Models — How ChatGPT and Its Peers Work

"The question is not whether machines can think, but whether people can recognize when machines are only pretending to." — Adapted from a common observation in AI research

Opening Vignette: Priya's Late Night

It's 11:47 p.m. on a Tuesday, and Priya is staring at a blank document. Her comparative politics essay is due in fourteen hours. She's done the readings — mostly — but her thesis feels mushy, and she can't figure out how to structure her argument about democratic backsliding in Hungary.

So she does what millions of students do now: she opens a chatbot.

"Help me outline an essay comparing democratic backsliding in Hungary and Poland," she types. Within seconds, the screen fills with a detailed outline — thesis statement included, topic sentences for each paragraph, even suggested counterarguments. It's... good. Really good. Better structured than what she'd been wrestling with for an hour.

But then she notices something. The chatbot cites a "2023 Freedom House report" that, when she checks, doesn't quite say what the AI claims it says. And one of the "scholars" it mentions — Dr. Elena Varga from the Budapest Institute for Democratic Governance — doesn't seem to exist. Priya Googles the name. Nothing. She tries different spellings. Still nothing.

The AI wrote with total confidence. It formatted the citation perfectly. It even included a page number. And it made the whole thing up.

Welcome to the world of large language models. They're the most impressive — and most misunderstood — AI systems most people will ever interact with. This chapter is about understanding what's actually happening behind the curtain, so that you can use these tools wisely and evaluate their outputs critically. We're not here to be dazzled or terrified. We're here to understand.

5.1 The Autocomplete on Steroids: Next-Token Prediction

Your Phone Already Does This (Sort Of)

You've used autocomplete. You start typing a text message — "Hey, are we still on for" — and your phone suggests "dinner," "tonight," or "tomorrow." That's a tiny prediction engine guessing what word comes next based on patterns it's seen before.

Now imagine scaling that up by a factor of a billion.

That's essentially what a large language model (LLM) does. It takes a sequence of text — everything you've typed so far in your conversation — and predicts the most likely next piece of text. Then it takes that prediction, adds it to the sequence, and predicts the next piece. And the next. And the next.

The key term here is next-token prediction. A token isn't exactly a word — it's a chunk of text that the model works with. Sometimes a token is a whole word ("democracy"), sometimes it's a word fragment ("demo" + "cracy"), and sometimes it's punctuation or a space. Most LLMs break text into tokens using a process called tokenization, which splits language into manageable, standardized pieces. The average English word is about 1.3 tokens.

Here's what matters: the model doesn't start with an idea it wants to express and then find words for it, the way you do when you write. Instead, it starts with your prompt and generates text one token at a time, each time asking: given everything so far, what token is most likely to come next?

Probability, Not Certainty

When the model predicts the next token, it doesn't pick just one answer. It calculates a probability distribution — a ranked list of every possible next token with a percentage chance attached to each. If the text so far is "The capital of France is," the model might assign:

• "Paris" → 97.2%
• "a" → 0.8%
• "located" → 0.6%
• "the" → 0.3%
• ... thousands more options trailing off toward 0%

Then the system samples from this distribution. Most of the time, it picks "Paris." But there's a small element of randomness built in, controlled by a setting called temperature. Low temperature means the model almost always picks the highest-probability token — safe, predictable outputs. High temperature means it's more willing to take chances, producing more creative (and sometimes more bizarre) text.

This is why you can ask the same question twice and get different answers. The model isn't retrieving a stored response. It's generating a new one each time, rolling the dice at every single token.

💡 Key Insight: LLMs don't "know" things the way an encyclopedia does. They calculate probabilities about what text should come next. This is a fundamentally different process from understanding, and keeping that distinction clear will serve you throughout this book.

Priya's Realization

Back to Priya. When the chatbot generated that fake citation — "Dr. Elena Varga, Budapest Institute for Democratic Governance, 2023, p. 47" — it wasn't lying. It wasn't trying to deceive her. It was predicting that, given the surrounding text about Hungarian democratic backsliding, a plausible-sounding citation with a Central European scholar's name and a specific page number was the most likely kind of text to appear next.

The model had seen thousands of academic essays with exactly that pattern. Name. Institution. Year. Page number. It reproduced the form of a citation flawlessly. But it had no mechanism for checking whether the content of that citation was real. It was filling in a pattern, not consulting a library.

This phenomenon has a name: hallucination. It's one of the most important concepts in this chapter, and we'll return to it. For now, just notice: the failure isn't random. It's a predictable consequence of how the system works. Pattern completion doesn't include fact-checking.

✅ Check Your Understanding: If someone says "ChatGPT knows the answer," what's more accurate to say about what's actually happening? How would you rephrase that claim using the concept of next-token prediction?

5.2 Attention Is All You Need (Simplified)

The Architecture That Changed Everything

In 2017, a team of researchers at Google published a paper with an unusually catchy title: "Attention Is All You Need." That paper introduced the transformer architecture, and it is not an exaggeration to say it changed the trajectory of AI research and, increasingly, of society.

Before transformers, language models processed text sequentially — one word at a time, left to right, like reading a sentence aloud. This worked, but it was slow, and the models struggled with long passages because they'd "forget" what was said at the beginning by the time they reached the end.

Transformers solved this with a mechanism called attention (or more precisely, self-attention). Here's the intuitive version.

The Dinner Party Analogy

Imagine you're at a dinner party with twelve people. In the old approach, you could only listen to the person directly next to you and try to pass messages down the table like a game of telephone. By the time a message reached the far end, it was garbled.

The transformer approach is different. Every person at the table can hear every other person simultaneously. But — and this is the crucial part — each person learns to pay more attention to the people who are saying things relevant to them. If you're discussing climate policy, you tune in to the environmental scientist three seats away and the economist across the table, while politely ignoring the person next to you who's talking about their cat.

That's self-attention. Every token in a sequence gets to "look at" every other token and decide how much each one matters for predicting what comes next. The word "it" in a sentence can look back and figure out whether "it" refers to "the policy," "the country," or "the research program" — not through grammatical rules, but by learning patterns of which words tend to relate to which other words across millions of examples.

Why This Matters for You

You don't need to understand the linear algebra behind transformers. But you do need to understand three things about this architecture:

1. It processes text in parallel, not sequentially. This is why modern LLMs are fast despite being enormous. They can look at all the tokens at once rather than plodding through them one by one.

2. It handles long-range dependencies. The model can connect a pronoun on page three to a noun introduced on page one. This is why LLMs can maintain something that feels like coherence across long conversations.

3. It's a general-purpose pattern finder. The transformer architecture isn't specific to language. Researchers have applied it to images, music, protein structures, and code. Its power comes from the attention mechanism's ability to find relationships in sequential data of almost any kind.

📊 By the Numbers: GPT-4 reportedly has over 1 trillion parameters — the internal numerical dials that get adjusted during training. GPT-3 had 175 billion. For perspective, the human brain has roughly 100 trillion synaptic connections. But be very careful with that comparison: a parameter and a synapse are not the same kind of thing, and the brain's architecture is fundamentally different from a transformer. The numbers are awe-inspiring, but they don't tell you as much as they seem to.

Layers, Layers, Layers

A transformer model isn't a single attention mechanism — it's many layers of them stacked on top of each other. Each layer captures different kinds of patterns. Early layers might learn basic grammar and word associations. Middle layers might capture more complex relationships — metaphor, tone, argument structure. Deep layers seem to encode something closer to abstract reasoning patterns, though "reasoning" is a word we need to be very careful with (more on that in Section 5.6).

Think of it like this: the first layer notices that "not" and "happy" often appear near each other. A middle layer learns that "not happy" often means something negative. A deeper layer learns that saying "I'm not unhappy" is different from "I'm unhappy" — it captures the double negative. Each layer builds on the one below it, creating increasingly sophisticated representations of what the text means.

Or at least, what the text is likely to be followed by. That distinction matters.

5.3 Pre-Training: Reading the Internet

How LLMs Learn Language

Building an LLM happens in stages. The first — and most expensive — stage is pre-training. This is where the model reads an enormous amount of text and learns the statistical patterns of language.

When we say "enormous," we mean it. Models like GPT-4 were trained on hundreds of billions of words drawn from books, websites, Wikipedia articles, code repositories, scientific papers, Reddit threads, news articles, and much more. The exact composition of these training datasets is often a closely guarded corporate secret, which raises its own set of concerns (we'll tackle those in Chapter 12).

During pre-training, the model's billions of parameters are gradually adjusted so that its predictions become more accurate. It sees a passage with the last few words hidden and tries to predict them. When it gets them wrong, the error signal flows backward through the network and nudges the parameters slightly — making that particular mistake a tiny bit less likely next time. This happens trillions of times.

The result is a model that has absorbed a staggering amount of information about how language works — not from grammar textbooks, but from sheer exposure. It knows that "The patient was diagnosed with" is likely to be followed by a medical condition. It knows that sonnets have fourteen lines. It knows that legal briefs use different vocabulary than sports commentary.

The Data Question

But here's the thing: the model learned from whatever was in the training data. And the internet is not a neutral, representative sample of human knowledge. It skews English. It skews Western. It overrepresents certain demographics, viewpoints, and types of content. It includes misinformation, toxic speech, outdated claims, and copyrighted material.

Remember the data chapter (Chapter 4)? The principle was: data is never neutral — it encodes the world that created it. That principle applies here with special force. An LLM trained on internet text absorbs the biases, blind spots, and errors of the internet.

If the training data contains more medical information about men than women, the model will be better at discussing men's health. If the training data overrepresents Western philosophical traditions, the model will be less fluent in non-Western perspectives. If the training data includes toxic stereotypes — and it inevitably does — traces of those stereotypes will lurk in the model's outputs.

⚠️ Content Warning / Limitation: Pre-training data necessarily includes harmful content. Model developers use filtering to remove the worst material, but no filter is perfect. This is not a bug that will be fixed in the next version — it's a structural challenge inherent to learning from human-generated text.

The Scale Problem

Pre-training a frontier LLM is extraordinarily expensive. Estimates for GPT-4's training cost range into the tens of millions of dollars, requiring thousands of specialized computer chips (GPUs) running for months. This has profound implications:

• Only a handful of organizations can afford to build frontier models. This concentrates power in ways that should concern anyone who cares about democratic governance of technology.
• The environmental cost is significant. Training runs consume enormous amounts of electricity. We'll explore this in detail in Chapter 18.
• Once trained, the model's knowledge is frozen. An LLM doesn't automatically learn about events that happened after its training data was collected. This is called the knowledge cutoff — and it means the model can confidently tell you about events that "happened" after its training ended, generating plausible but potentially fabricated details.

5.4 Fine-Tuning and RLHF: Teaching Manners

From Raw Model to Helpful Assistant

If you could interact with a pre-trained model before any additional training, you'd find it... strange. It can complete text beautifully, but it doesn't really answer questions. Ask it "What's the capital of France?" and it might continue the text as if it's a quiz show: "What's the capital of France? What about Germany? Test your knowledge of European capitals!" It learned to predict text, not to be helpful.

This is where fine-tuning comes in. Fine-tuning takes the pre-trained model and trains it further on a smaller, carefully curated dataset — typically examples of the kind of behavior the developers want. For a chatbot, this means thousands of examples of questions paired with helpful, accurate, appropriate answers.

Think of pre-training as giving someone a massive education — they've read millions of books and absorbed an incredible range of knowledge. Fine-tuning is like then giving that person a specific job and training them on workplace expectations. "When a customer asks a question, respond directly. Don't go off on tangents. Be polite."

RLHF: Learning from Human Feedback

One of the most important innovations in making LLMs useful was Reinforcement Learning from Human Feedback (RLHF). Here's the basic idea:

1. The model generates several possible responses to a prompt.
2. Human evaluators rank those responses from best to worst.
3. The model is trained to produce more responses like the highly ranked ones and fewer like the poorly ranked ones.

This process is what transforms a raw text-prediction engine into something that feels like a helpful conversational partner. The humans doing the ranking encode preferences: be truthful, be helpful, don't produce harmful content, acknowledge uncertainty, refuse inappropriate requests.

But notice what's happening here. The model isn't learning to be truthful in any deep sense. It's learning that responses which appear truthful tend to get higher rankings. It's learning to produce text that looks like what a helpful, honest assistant would say. This is an important distinction — one we'll return to in Section 5.6.

The People Behind the Curtain

RLHF requires a lot of human labor. Thousands of workers — many of them contractors in lower-income countries — spend their days reading and ranking AI outputs. Some of this work involves reviewing deeply disturbing content: the toxic, violent, or hateful text that the model needs to learn to avoid.

Investigations have revealed that some of this labor is performed under difficult conditions: low pay, limited mental health support, and exposure to traumatic content. When you use a chatbot that politely declines to generate hate speech, that safety feature was built on human labor — and it's worth asking who did that labor and how they were treated.

🔗 Connection to Chapter 4: Remember how training data encodes human choices and biases? RLHF adds another layer: the preferences of the human evaluators also shape the model. Their cultural backgrounds, their employer's guidelines, their personal judgments about what counts as "helpful" or "harmful" — all of it influences what the model learns. The AI is, once again, a reflection of the people and processes that built it.

Constitutional AI and Other Approaches

RLHF isn't the only approach. Some developers use a technique called Constitutional AI (CAI), where the model is given a set of written principles — a "constitution" — and trained to evaluate and revise its own outputs against those principles. Other approaches use automated feedback from other AI models rather than relying entirely on human evaluators.

These are active areas of research, and the methods will continue to evolve. The durable insight is this: the behavior of a deployed LLM is not just a product of its training data. It's shaped by deliberate choices about what counts as good output, made by specific people and organizations with their own values and incentives.

5.5 What LLMs Can Do (and What They Can't)

The Impressive List

Let's be honest about what's genuinely remarkable. Modern LLMs can:

• Write coherent, well-structured text across a stunning range of genres: essays, code, poetry, emails, legal documents, recipes, dialogue, technical manuals
• Translate between languages with quality that rivals many professional translators for common language pairs
• Summarize long documents and extract key information
• Answer questions about a huge range of topics, often with impressive accuracy
• Generate code in dozens of programming languages, debug existing code, and explain how code works
• Engage in extended conversations that maintain context and coherence
• Adapt their tone and style based on instructions — formal, casual, persuasive, explanatory

This is genuinely impressive. Five years before the launch of ChatGPT, most AI researchers would not have predicted that language models would achieve this level of fluency this quickly.

The Sobering List

But LLMs also systematically struggle with — or outright fail at — several important tasks:

Factual accuracy. LLMs hallucinate. They generate text that is confident, fluent, and wrong. They invent citations, fabricate statistics, attribute quotes to people who never said them, and describe events that never happened. This isn't occasional; it's structural. A model that predicts likely text will sometimes predict text that sounds true but isn't.

Mathematical reasoning. Despite being able to discuss mathematical concepts fluently, LLMs frequently make arithmetic errors and struggle with multi-step mathematical reasoning. They can explain what a derivative is but may botch a straightforward calculus problem. (Newer models are improving here, but the underlying architecture was not designed for reliable computation.)

Temporal awareness. LLMs don't know what day it is. They don't know what happened after their training cutoff. They can't tell you today's weather or yesterday's stock prices unless they have access to external tools — and even then, they need to be specifically engineered to use those tools reliably.

Consistent reasoning across long contexts. While transformers handle long-range dependencies better than older architectures, LLMs can still lose track of details in very long conversations or documents. They may contradict themselves, forget constraints you specified earlier, or gradually drift from your original request.

Understanding vs. pattern matching. This is the big one, and it deserves its own section.

💡 Myth vs. Reality

Myth: "The AI understood my question and gave me an informed answer." Reality: The AI processed your text, identified patterns matching its training data, and generated a statistically plausible continuation. It may have produced accurate information, but not because it understood either your question or its answer.

Myth: "LLMs are getting smarter every generation — they'll be truly intelligent soon." Reality: LLMs are getting better at producing text that appears intelligent. Whether that constitutes "real" intelligence is a philosophical question that the technology alone cannot answer. Bigger models make fewer obvious errors, but they don't resolve the fundamental gap between pattern matching and understanding.

Priya Adjusts Her Approach

After the fake citation incident, Priya doesn't stop using the chatbot. But she changes how she uses it. She starts treating it as a brainstorming partner — great for generating outlines, suggesting counterarguments, and helping her organize her thoughts. But she fact-checks every specific claim. She verifies every citation. She uses it to strengthen her thinking, not replace it.

"It's like having a really smart study buddy who sometimes makes things up with a completely straight face," she tells her roommate. "Useful, but you have to stay alert."

That's a pretty good working relationship with an LLM.

✅ Check Your Understanding: Name two tasks that LLMs do well and two they struggle with. For each struggle, explain why the limitation exists given how LLMs work (next-token prediction on training data).

5.6 The Stochastic Parrot Debate: Understanding vs. Mimicry

The Question That Won't Go Away

In 2021, a group of researchers — Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (writing as "Shmargaret Shmitchell" after her departure from Google) — published a paper that has become one of the most cited and debated in modern AI. Its title asked whether large language models are "stochastic parrots": systems that produce language without understanding it, the way a parrot might repeat phrases without grasping their meaning.

The word stochastic just means "involving randomness" — which connects directly to the temperature and probability concepts we discussed in Section 5.1. A stochastic parrot, then, is a random-pattern repeater. It produces human-sounding language by recombining patterns from its training data, with some randomness mixed in. It doesn't mean what it says. It doesn't believe its claims. It's performing language, not using it.

The Other Side

Not everyone agrees with this characterization. Some researchers and philosophers argue that:

• The line between "mimicry" and "understanding" may not be as clear as we think. If a model can pass medical exams, write poetry that moves people, and explain complex concepts in multiple ways, at what point does sophisticated pattern matching become indistinguishable from understanding? (This echoes the philosophical thought experiment of the "Chinese Room," proposed by philosopher John Searle in 1980.)

• Emergent capabilities appear in larger models. As models get bigger, they sometimes develop abilities that weren't explicitly trained. They can do chain-of-thought reasoning, solve novel analogies, and transfer skills between domains. Does this suggest something beyond mere pattern matching?

• Human language use might be more pattern-based than we'd like to admit. When you use an idiom, tell a familiar joke, or structure an argument using a template you learned in school, how different is that from what the model does?

A Framework for Thinking About It

Rather than trying to resolve this philosophical debate — which may ultimately be irresolvable with our current understanding — here's a practical framework:

Notice that you don't need to answer "Does the AI truly understand?" to use this framework effectively. Whether or not the model "understands" your question, you still need to verify its outputs. Whether or not it's "just" a stochastic parrot, its outputs can still cause real harm. The practical questions are answerable even when the philosophical ones aren't.

🧪 Thought Experiment: Imagine a student submits an essay that is beautifully written, well-argued, and entirely factually accurate. Later, you learn it was generated by an LLM. Has the student learned anything? Does the essay demonstrate understanding? Now imagine a different student writes a messier essay with some errors but shows genuine engagement with the material. Which student do you think learned more? What does your answer reveal about what you value in education — the product or the process?

The Threshold Concept

Here it is — the idea that, once you really grasp it, changes how you evaluate every AI claim you encounter:

🔴 Threshold Concept: LLMs predict the next word — they don't understand meaning. This doesn't mean their outputs are useless. It means their outputs are generated through a process that has no mechanism for truth, understanding, or intent. The text may be accurate. It may be helpful. But its accuracy is incidental to the generation process, not guaranteed by it. Every output requires human evaluation.

Let this sink in. The model doesn't know whether it's telling you the truth. It doesn't know what "truth" is. It's producing the most probable next token given everything that came before. Sometimes that process yields profound insights. Sometimes it yields confident nonsense. The process itself can't tell the difference.

This is the capability vs. understanding theme that runs throughout this book. The capability is real and impressive. The understanding is absent — or at least, absent in any sense of the word that we currently have good reason to attribute to these systems.

What This Means for ContentGuard

Remember ContentGuard, the social media content moderation system from Chapter 1? If ContentGuard uses an LLM to evaluate whether a post violates platform guidelines, the model is doing pattern matching: "Posts that look like this have been flagged as harmful." It's not understanding the nuance of satire, cultural context, or the difference between reporting violence and glorifying it. It's predicting whether the post resembles the kind of content that human moderators have previously flagged.

This works reasonably well at scale — much better than keyword-based filters. But it fails in exactly the cases where nuance matters most. And those are often the cases with the highest stakes: political speech, minority communities using reclaimed language, artistic expression that deals with difficult themes.

5.7 Chapter Summary

Let's bring it together.

Large language models are next-token prediction engines built on the transformer architecture. They process text by calculating which token is most likely to come next, given everything that precedes it. They do this using attention mechanisms that allow every part of the input to influence every other part, enabling them to capture complex patterns across long stretches of text.

They learn through stages. Pre-training on massive datasets gives them broad language ability. Fine-tuning on curated examples shapes their behavior for specific tasks. RLHF and similar techniques teach them to produce outputs that humans rate as helpful, harmless, and honest.

They are remarkably capable. They write, translate, summarize, code, and converse at a level that would have seemed like science fiction a decade ago.

They are also systematically limited. They hallucinate. They can't verify their own claims. They reflect the biases in their training data. They don't update their knowledge in real time. And the question of whether they truly "understand" anything remains genuinely open.

The practical takeaway is clear: Use LLMs as powerful tools, not as authoritative sources. Verify their outputs. Understand their limitations. And maintain a healthy skepticism about claims — from either enthusiasts or critics — that outrun the evidence.

Priya figured this out the hard way at 11:47 p.m. on a Tuesday. You can figure it out right now, while the stakes are lower.

🐍 Optional Code: Your First LLM API Call (Conceptual)

This section is entirely optional and requires no programming background to understand.

If you were to interact with an LLM through code rather than a chat interface, the basic structure would look something like this:

# Conceptual example — shows the structure of an LLM API call
# You would need an API key and the openai library installed to run this

from openai import OpenAI
client = OpenAI()  # uses your API key

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "What is democratic backsliding?"}],
    temperature=0.7  # Controls randomness: 0 = predictable, 1 = creative
)

print(response.choices[0].message.content)

Notice the temperature parameter — that's the randomness dial from Section 5.1. At 0.0, the model would give you the most predictable answer. At 1.0, it would take more risks. Most applications use something in the middle.

The point isn't to teach you to code. It's to demystify: underneath the chat interface, an LLM is just a function that takes text in and produces text out, one token at a time.

AI Audit Report — Chapter 5 Checkpoint

📋 Project Component: If your chosen AI system uses LLMs, describe how; if not, explore how LLMs could augment it.

Add the following to your ongoing AI Audit Report:

1. LLM Usage: Does your chosen system use a large language model? If yes, describe what role it plays (e.g., generating text, interpreting user queries, classifying content). If no, describe how an LLM could be integrated and what it would add.

2. Hallucination Risk: Given what you learned about next-token prediction, where might hallucination be most dangerous in your system? What safeguards would you want in place?

3. Training Data Questions: What training data might your system's LLM (actual or hypothetical) have been trained on? What biases or gaps might that data contain?

4. The Understanding Question: Does your system need its AI to understand its domain, or is pattern matching sufficient for the task? Explain your reasoning.

Spaced Review: Looking Back

Before moving on, revisit these ideas from earlier chapters:

🔵 From Chapter 1: You learned that AI is a spectrum of techniques, not a single technology. How does the LLM fit into that spectrum? Is it the "most AI" kind of AI, or just one approach among many?

🔵 From Chapter 2: You explored the history of AI development. Where do transformers and LLMs fit in the timeline of AI progress? What earlier approaches did they build on or replace?

🔵 From Chapter 3: The threshold concept was "machines learn from patterns in data, not from understanding." How does that concept apply specifically to LLMs? Has your understanding of that idea deepened or changed after this chapter?

Key Terms Introduced in This Chapter