Chapter 2: How Language Models Think: A Conceptual Framework

You do not need to be a machine learning engineer to use AI tools well. But you do need to understand, at a conceptual level, what is actually happening when you type a message and receive a response. Without that understanding, you are working in the dark — unable to predict failures, unable to diagnose problems, unable to improve beyond a certain ceiling.

This chapter is not a technical deep-dive. There is no mathematics here. What you will find instead is a precise, accurate conceptual framework for how language models process information and generate text — the kind of framework that makes behavior predictable and, by extension, controllable.

By the end of this chapter, you will understand why AI tools are confidently wrong, why they do not remember your previous conversations, why the same prompt can produce different outputs on different days, and why they have hard limits on how much they can consider at once. You will understand these things not as quirks to work around but as logical consequences of how these systems are built.

That understanding is the foundation of everything that follows.

2.1 The Building Block: What Is a Token?

Before anything else, you need to understand tokens — because everything about how language models work is built on top of this concept, and most people who use AI tools daily have only a vague sense of what tokens actually are.

A token is not a word. It is roughly a piece of a word, though it can also be a whole word, a punctuation mark, a space, or a fragment of a longer word. The way a model like GPT-4 or Claude processes language is by breaking text into these small units before doing anything else with it.

To understand why, think about how a model learns. It processes enormous amounts of text — we will get to training in a moment — and it needs to build a vocabulary of meaningful units. If every unique word was a separate unit, you would have hundreds of thousands of vocabulary entries, and rare words (technical terms, proper nouns, foreign words) would be underrepresented. Instead, models use subword tokenization: common words become single tokens, while longer or rarer words are split into recognizable fragments.

The word "understanding" might be one token. The word "tokenization" might be split into "token" and "ization." The word "antidisestablishmentarianism" would likely become several tokens. Punctuation, spaces, and newlines are also tokens. A rough rule of thumb: in English, one token corresponds to about three to four characters of text, or about three-quarters of a word on average.

💡 Intuition: Think of tokens as the model's "alphabet" — but where our alphabet has 26 letters, a model's vocabulary has tens of thousands of tokens. Every piece of text you send is first converted into a sequence of these tokens before the model processes it.

Why Tokens Matter for You

The token concept has three concrete implications for how you work with AI:

Context windows are measured in tokens. When we talk about a model's context window — how much it can "hold in mind" at once — we are talking about a maximum number of tokens. A model with a 128,000-token context window can process roughly 96,000 words in a single session. This sounds like a lot until you are working with very long documents, code repositories, or extended conversations.

Costs scale with tokens. If you are using AI tools via an API and paying per use, your costs are almost always measured in input tokens (what you send) and output tokens (what you receive). Understanding tokenization helps you estimate costs and make deliberate choices about what to include in your prompts.

Some things cost more tokens than you expect. Code, for example, tends to be denser in tokens than prose — special characters, indentation, and syntax all add up. Long lists of instructions, detailed system prompts, and extensive examples all consume your context window budget. This matters when you are working at scale or near the limits of a model's context.

⚠️ Common Pitfall: People often assume that "context window" refers to the visible conversation history in their interface. It does not — it refers to all tokens the model processes at once, including any system instructions, document chunks, or retrieved content that happens behind the scenes in a tool. You may have less usable context than you think.

2.2 Next-Token Prediction: The Fundamental Mechanism

Here is the most important conceptual fact about language models: at their core, they do one thing. They predict what token should come next, given all the tokens that came before.

That is it. That is the fundamental mechanism. Everything else — the apparent reasoning, the coherent paragraphs, the code that works, the explanations that illuminate — emerges from this single operation performed billions of times across trillions of training examples.

Let that sit for a moment, because it is simultaneously simpler than most people expect and weirder than most people appreciate.

When you type "The capital of France is," the model predicts that the next token should be something like "Paris." But not because it looked up a fact in a database. Because in its training data, texts containing that exact sequence of words overwhelmingly followed with "Paris." The model has learned that this pattern is extremely probable.

This extends to much more complex cases. When you ask a model to explain a concept, it generates an explanation not by retrieving an explanation from storage, but by predicting, one token at a time, what sequence of tokens constitutes a plausible explanation of that concept given everything it learned during training.

What "Plausible" Means Here

The key word is plausible, not correct. The model is optimizing for generating text that follows the statistical patterns of the training data — text that looks and reads like something a knowledgeable person might write about the topic. In many cases, plausible and correct coincide perfectly. In many other cases, they diverge in subtle or not-so-subtle ways.

💡 Intuition: Imagine an extraordinarily well-read person who has absorbed millions of books, articles, and documents. They have never experienced anything directly — they have only read about everything. When you ask them a question, they compose an answer based entirely on the patterns of how that question has been answered in all the texts they have read. Most of the time, those patterns lead to good answers. Sometimes they lead to confident-sounding nonsense.

This is sometimes called the "fluency-accuracy gap" — the model's ability to produce fluent, well-structured, confident-sounding text is not tightly coupled to the accuracy of that text. We will return to this point, because it is one of the most practically important things to understand about working with AI tools.

The Generation Process

When a model generates a response, it does not write the whole thing and then show it to you. It generates one token at a time, in sequence, with each new token being predicted based on everything that came before it (your prompt, plus all the tokens it has generated so far). This is why output "streams" in most interfaces — you are watching the generation happen in real time.

This sequential, probabilistic generation has important implications. The model cannot go back and revise a token it has already committed to — it can only continue forward. This is one reason why asking a model to "think step by step" before answering can improve output quality: it prompts the model to generate intermediate reasoning tokens that then become part of the context for generating the final answer.

✅ Best Practice: For complex reasoning tasks, explicitly prompt the model to show its work before giving a final answer. This is not just pedagogically useful — it actually changes what the model generates, because the intermediate steps become part of its own context.

2.3 Training: What the Model Learned From

Language models learn from text. The training process involves exposing the model to enormous amounts of text — web pages, books, code repositories, academic papers, news articles, forum discussions, and much more — and having it repeatedly perform next-token prediction on that text, adjusting its internal parameters to get better at the task over time.

This process produces a model that has absorbed, in a statistical sense, an enormous amount of human knowledge, reasoning patterns, linguistic structure, and domain-specific content. The result is something that can discuss almost any topic in a coherent way, code in dozens of programming languages, write in various styles, translate between languages, and perform a remarkable range of reasoning tasks.

But training has limits that are crucial to understand.

The Training Cutoff

Every language model has a training cutoff — a date after which it has no information about the world. The model was trained on data collected up to that date, and everything that happened afterward is simply absent from its knowledge.

This is not a bug. It is a fundamental consequence of how training works. You cannot train on data that does not exist yet, and keeping a model continuously updated in real time is technically and computationally prohibitive (though some systems approximate this through retrieval mechanisms — more on that later).

The training cutoff creates what we might call the "frozen knowledge" problem. The model knows the world as it was at a particular moment in time, but it will discuss events, technologies, and facts using the confident, present-tense language of a knowledgeable person — without any indication that its knowledge might be months or years out of date.

⚖️ Myth vs. Reality

Myth: If you ask an AI a question about something after its cutoff, it will tell you it does not know.

Reality: The model may give you a confident, plausible-sounding answer drawn from patterns in its pre-cutoff training data — an answer that is outdated or simply wrong. It does not know it does not know. The absence of information is not the same as knowing there is a gap.

What This Means in Practice

If you ask a language model about a Python library that released a major update after its training cutoff, it will describe the library as it existed before that update — with the same confidence it would use to describe a timeless mathematical principle. If you ask it about a company's current products, pricing, or leadership, you may get information that was accurate eighteen months ago. If you ask it about recent events, research, or regulatory changes, the same problem applies.

The model does not flag this automatically. It does not know there has been a cutoff. It knows what it knows, and it does not know what it does not know.

⚠️ Common Pitfall: One of the most dangerous behaviors is asking an AI about "current best practices" in a fast-moving field. The model will give you confident, well-organized advice that may be genuinely outdated. Always verify time-sensitive information through primary sources.

Raj, who uses GitHub Copilot and other AI tools for development work, learned this the hard way. We explore the full story in Case Study 2.1: "The Frozen Clock Problem." The short version: he used an LLM to research a third-party API, received a detailed and confident explanation of how to authenticate and make requests — and the explanation was accurate for the version of the API that predated the cutoff. The current version had introduced breaking changes. He shipped code based on the AI's description. The code did not work. We will examine exactly how this happened and how it could have been avoided.

2.4 Temperature and Sampling: Why Outputs Vary

You may have noticed that if you send the same prompt to a language model twice, you often get different responses. Sometimes the responses are similar in substance but different in phrasing. Sometimes they differ more substantially. This is not a malfunction. It is a designed feature — and understanding why it exists helps you work with it deliberately.

When a model predicts the next token, it generates a probability distribution: essentially a ranked list of all possible next tokens, each with an associated probability. The highest-probability token is the one the model considers most likely given the context. But in most implementations, the model does not always pick the highest-probability token. Instead, it samples from the distribution — sometimes choosing a lower-probability option.

This is controlled by a parameter called temperature.

At low temperature (approaching zero), the model consistently picks the highest-probability token. The output becomes deterministic and repetitive — predictable, but also rigid and sometimes monotonous.

At high temperature, the model samples more liberally from lower-probability options. The output becomes more varied, more creative, and more surprising — but also potentially less coherent, more prone to tangents, and more likely to produce unexpected results.

Most practical applications use a temperature somewhere in the middle, balancing coherence with variety.

💡 Intuition: Temperature is like a creativity dial. Turn it down, and the model produces the most expected response — the response that most closely follows the patterns it has seen most often. Turn it up, and it takes more creative risks — exploring less-trodden paths through the space of possible text.

Why This Matters for Your Work

Understanding temperature has two practical implications:

Expect variation. If you are using a tool with default settings, your results will naturally vary between runs, even with identical prompts. For tasks where consistency is critical — generating standard documents, following a strict format, producing repeatable outputs — you may want to explore whether the tool you are using allows you to reduce temperature.

Use variation strategically. When you want exploratory thinking, multiple angles on a problem, or creative options, high-temperature settings are your friend. Running the same prompt multiple times and comparing outputs can surface a range of approaches you would not get from a single run.

✅ Best Practice: For analytical or factual tasks, prompt the model to think step by step (which tends to constrain randomness through logical dependencies). For creative or generative tasks, consider multiple runs to get a range of options.

There are also other sampling strategies beyond temperature — techniques called top-p sampling and top-k sampling that constrain which tokens the model can choose from — but the conceptual point is the same: language model outputs are probabilistic, not deterministic, and that has real consequences for how you work with them.

2.5 Context Windows: What the Model Can Hold in Mind

The context window is one of the most practically important concepts for working with AI tools, and also one of the most frequently misunderstood.

Every language model has a maximum context window — a limit on the total number of tokens it can process in a single inference. Everything the model can "consider" when generating a response must fit within this window: your prompt, the conversation history, any documents you have provided, any system instructions running in the background, and the response it has generated so far.

When content exceeds the context window, it must be handled somehow — either truncated (the oldest content is dropped), summarized (a compressed version is substituted), or split across multiple calls. Different tools handle this differently and often invisibly.

Why This Matters for Long Conversations

Elena, the freelance consultant, encountered this problem directly. She had been working with a language model on a large project brief, refining strategy across a two-hour conversation. The early parts of the conversation had established important constraints and decisions. As the conversation grew longer, something strange happened: the model's advice started to contradict those earlier decisions. It was not being inconsistent in its reasoning — it was simply unable to see the earlier parts of the conversation anymore. They had fallen outside the context window.

This is not a hypothetical edge case. It is a common experience for anyone who works with AI tools in extended sessions. The full story of what happened to Elena, how she diagnosed it, and what she changed about her workflow is in Case Study 2.2: "Elena and the Vanishing Memory." We will explore the context window problem in depth there.

Practical Implications

Long documents need careful handling. If you are working with documents that approach or exceed the context limit, the model may not be able to consider the entire document simultaneously. Different tools handle this differently — some use chunking and retrieval, some simply truncate — and the behavior is rarely transparent.

Context window does not equal memory. Within a session, the model maintains context through the conversation history stored in the context window. But this is not "memory" in any meaningful sense — it is just text the model can currently see. When a session ends, the context is gone. The next session starts with an empty context.

Earlier content gets displaced. In a very long conversation, content you provided early — constraints, preferences, background information — may eventually be pushed out of the context window by later content. You cannot always see when this happens, which makes it insidious.

⚠️ Common Pitfall: People often assume that because they can scroll up to see earlier parts of a conversation in the interface, the model can also "see" that content. This is not necessarily true. The interface may show you the full history while the model is only processing the most recent portion.

✅ Best Practice: For extended projects or long documents, periodically re-anchor the model by summarizing key constraints, decisions, and context at the start of each new session or major conversation segment. Do not rely on the model to remember what you established hours ago.

2.6 The Fluency-Accuracy Gap: Why AI Is Confidently Wrong

We touched on this earlier, but it deserves its own treatment because it is the failure mode that causes the most real-world harm for people using AI tools.

Language models are extraordinarily good at producing fluent, well-structured, confident-sounding text. They have been trained on the best writing humanity has produced — not because anyone selected for quality, but because quality writing tends to be overrepresented in the sources that end up in training data (published books, academic papers, quality journalism). The result is a system that can produce prose that reads as knowledgeable and authoritative across virtually any topic.

This fluency is not coupled to accuracy. The model produces text that sounds like what a knowledgeable person would say, but it has no internal truth-checking mechanism that validates whether what it is saying is actually true. When it generates a false statement — whether due to outdated training data, pattern-matching on related-but-not-exact material, or what are technically called hallucinations — it does so with exactly the same confident tone as when it generates a true one.

Hallucinations

The term "hallucination" in the context of AI refers to the generation of plausible-sounding but factually incorrect content. The model might cite a paper that does not exist, give the wrong date for a historical event, attribute a quote to the wrong person, or describe a feature of a software library that was never part of that library.

These are not lies. The model has no concept of truth and falsehood in the way humans do — it is not deliberately deceiving you. It is generating the next most probable token based on patterns in its training data, and sometimes those patterns lead it to produce incorrect content.

⚖️ Myth vs. Reality

Myth: AI tools only confabulate on obscure topics where they have little data.

Reality: Hallucinations occur across domains, including well-represented ones. A model might give you a confident but wrong answer about a well-known library, a major historical figure, or a basic scientific fact — not because it lacks data on the topic, but because its pattern-matching produced a plausible-but-false token sequence. Familiarity of subject is not a reliable proxy for accuracy.

The Practical Challenge

The challenge is that the fluency of the output makes errors hard to spot. If a model generates an explanation that is 95% accurate and 5% wrong, the 5% does not usually announce itself. It is embedded in prose that reads as authoritative. The reader who does not know the subject well enough to catch the error will not catch it.

This is why trust calibration — which we explore deeply in Chapter 4 — is so essential. The appropriate response is not to distrust AI output entirely, which would eliminate enormous value. The appropriate response is to develop domain-specific instincts for what types of claims require verification.

📊 Research Breakdown: Studies of language model hallucination have found that accuracy rates vary significantly by task type. Factual recall tasks (asking for specific facts) tend to show higher hallucination rates than reasoning tasks (asking the model to work through a problem). However, the correlation between model confidence and model accuracy is weak — models do not reliably express less confidence when they are more likely to be wrong. This means that surface-level confidence in output is not a reliable accuracy signal.

2.7 Emergent Capabilities: What No One Planned For

One of the genuinely surprising findings from the development of large language models is that certain capabilities appear to emerge as models scale up — capabilities that were not explicitly trained for and were not predicted in advance.

At small scale, a model might be able to perform simple pattern-matching tasks. At medium scale, it might show some reasoning ability. At very large scale, it might demonstrate capabilities that look like multi-step planning, analogical reasoning, and even rudimentary theory of mind — the ability to model what another person knows or believes.

These capabilities are called emergent because they arise from scale rather than explicit design. No one built in a "reasoning module." The reasoning appears to emerge from learning to predict text at sufficient scale.

💡 Intuition: An analogy: if you teach a system to mimic how humans write, and humans write in ways that express their reasoning, then a system that perfectly mimics human writing would also appear to reason. Emergence may be the model learning to reproduce patterns of human reasoning as expressed in text, rather than developing an internal reasoning process analogous to human cognition.

What This Means for Understanding AI Capability

Emergent capabilities make language models hard to characterize. Their capabilities are not a fixed list. As models scale and as training methods improve, new capabilities appear. This means that intuitions formed about what a particular generation of AI can and cannot do may not apply to the next generation.

It also means that the capabilities are uneven and not always predictable. A model might demonstrate sophisticated reasoning on one type of problem and fail on a structurally similar problem that requires slightly different pattern-matching. This is sometimes called "capability brittleness" — the capabilities are real but not robust across all variations of a task.

⚠️ Common Pitfall: After watching a model handle a complex task brilliantly, it is tempting to assume it can handle all similar tasks at the same level. Emergent capabilities are uneven. A model that reasons beautifully through one problem may fail on a seemingly simpler variant. Always test on your actual task rather than inferring from performance on related tasks.

2.8 What "Thinking" Means for an LLM

There is an important question lurking beneath everything in this chapter, and it deserves direct treatment: when a language model appears to reason, understand, or know something, what is actually happening?

The honest answer is that we do not fully know, and this is one of the most active areas of debate in AI research. What we can say with confidence is that what language models do is fundamentally different from human thought, even when the outputs look similar.

Human thinking involves: - Continuous experience of the world - Working and long-term memory across time - Embodied perception and action - Emotional and motivational states - A persistent self-model

Language model "thinking" involves: - Processing a sequence of tokens in a context window - Pattern-matching against statistical regularities learned from training - No persistence between sessions - No experience, embodiment, or motivation - No self-model in any meaningful sense

When a model "understands" your question, it processes the token sequence and generates a response based on statistical patterns. This can look indistinguishable from understanding. Sometimes it produces outputs that are indistinguishable from genuine comprehension. But the mechanism is categorically different.

The "Brilliant Student Who Read Everything" Analogy

One of the most useful ways to think about a language model is as a student who has read an enormous amount — more than any human could read in a lifetime — but who has never directly experienced anything. They can tell you how it feels to grieve because they have read thousands of descriptions of grief. They can explain how to change a tire because they have read the instructions many times. They can describe the smell of rain because they have encountered that description repeatedly.

But they have never felt grief, changed a tire, or smelled rain. When their descriptions are good, it is because the descriptions in their training data were accurate. When they make errors, it is often because they are pattern-matching on related content rather than accessing genuine understanding.

This analogy is imperfect, as all analogies are. But it captures something important: the appearance of knowledge and the mechanism behind it are quite different, and that difference has practical implications for how you work with these tools.

💡 Intuition: The model is not thinking through your problem the way a human collaborator would. It is producing the most statistically plausible response to your prompt. Very often, that response is genuinely helpful and accurate. Sometimes, it is fluently wrong. Knowing which is which requires the kind of domain knowledge and critical thinking that only you can bring.

2.9 Raj's Scenario: When Copilot Fails at Certain Tasks

Raj has been using GitHub Copilot for about two years. He is sophisticated about it in many respects — he has learned which kinds of suggestions to trust and which to verify, and he has developed good habits around code review. But he still encounters failures that confuse him, and understanding the mechanics of how language models work helps explain most of them.

Why Copilot fails at novel architectures. Copilot is trained on code from public repositories. If you are building something that uses a pattern that is well-represented in that training data — a REST API in Express.js, a Django model, a React component — Copilot performs well. If you are using a framework, library version, or architectural pattern that is underrepresented in its training data, suggestions become less reliable. Not wrong in an obvious way, but subtly misaligned with your specific context.

Why recent library versions trip it up. Libraries evolve. If a library released a major update after Copilot's training cutoff — changing function signatures, deprecating methods, restructuring its API — Copilot may generate code that targets the old version. It has no way to know the library has changed. The suggestions look right because they follow the patterns of the pre-update version; they fail because the runtime environment has the new version.

Why complex reasoning tasks produce inconsistent results. Code completion tasks (filling in what comes next based on context) are the sweet spot for Copilot — they are closest to the next-token prediction task the model was trained on. Higher-level reasoning tasks (how should I structure this entire feature? What's the right data model for this problem?) are further from that core training signal. The model can generate plausible-looking answers, but the quality varies considerably.

Why it sometimes gets confident about wrong things. The fluency-accuracy gap shows up in code too. A suggestion might look syntactically correct, use the right variable names, and even have a comment explaining what it does — and still contain a logical error. Fluency does not imply correctness in code any more than in prose.

Understanding these mechanics, Raj can anticipate where Copilot is likely to fail and apply more scrutiny in those situations — while trusting it more confidently in situations where its training signal is strong and recent.

2.10 Elena's Scenario: The Memory Problem

Elena works on multi-week consulting projects. Her workflow involves using AI tools extensively for research synthesis, document drafting, and strategic thinking. She has developed a sophisticated set of practices, but one problem kept recurring: the model would give advice in later parts of a conversation that contradicted decisions made earlier.

At first she attributed this to model inconsistency. After learning about context windows, she understood what was actually happening.

Her working sessions with an AI tool were long — sometimes lasting several hours with hundreds of exchanges. The early parts of these conversations established critical context: the client's constraints, the strategic direction they had agreed on, the specific framing of the problem. As the conversation grew, that early context was scrolling out of the active context window. By the time she was deep in the tactical details, the model could no longer see the strategic framing from the beginning.

What Elena changed. She developed two practices that dramatically reduced this problem:

First, she began each session with a structured context document — a brief summary of the project's key parameters, constraints, and established decisions — that she included at the start of every conversation. This ensured the model always had access to critical context, even when it could no longer see earlier parts of the conversation.

Second, she learned to recognize the symptoms of context dropout — moments when the model's advice seemed to ignore previously established constraints. Rather than getting frustrated, she now treats these moments as a diagnostic signal and re-anchors by restating the relevant context.

Elena's situation illustrates a principle that will come up repeatedly in this book: the model does not maintain a model of your project, your client, or your ongoing relationship with it. Every session is a fresh start, and even within a session, the context window is finite. Working effectively with AI means taking responsibility for context management in a way that human collaboration does not require.

2.11 Putting It Together: A Framework for Prediction

Everything in this chapter connects to a single practical capability: the ability to predict how a language model will behave in a given situation, and to adjust your approach accordingly.

Here is a simplified decision framework that draws on everything you have learned:

Before you send a prompt, ask:

1. Is this time-sensitive information? If so, the training cutoff may be relevant. Plan to verify.

2. How much context does this task require? If the task depends on information from earlier in a long conversation, consider whether that information is still in the context window — or re-provide it.

3. Is this a pattern-completion task or a reasoning task? Pattern-completion tasks (writing in a specific style, generating code for a common framework, summarizing a document) are typically more reliable than open-ended reasoning tasks. Calibrate your trust accordingly.

4. Am I asking for something that requires real-world experience or specialized expertise? The "brilliant student who read everything" is excellent at synthesizing what others have written. They are less reliable on things that require direct experience or current specialized knowledge.

5. Do I need consistent output, or is variation acceptable? If consistency matters, consider whether to adjust temperature settings or explicitly constrain the format in your prompt.

After you receive a response, ask:

1. Does this sound fluent and authoritative? Remember that fluency is not accuracy. Sound confident output still needs to be evaluated.

2. Are there specific claims here that are verifiable and consequential? Verify them.

3. Does the response seem to reflect the context I established? If it ignores constraints or contradicts earlier decisions, context dropout may be the cause.

✅ Best Practice: Develop the habit of mentally running through this framework before high-stakes prompts. For casual, exploratory use, the overhead is not worth it. For anything consequential — code that will ship, advice you will share with clients, analysis that will inform decisions — this brief audit is valuable.

Chapter Summary

Language models work by predicting the next token in a sequence, one token at a time, based on patterns learned during training on enormous amounts of text. This mechanism produces remarkably capable systems that can discuss almost any topic with apparent intelligence — and also produces characteristic failure modes that follow directly from the mechanism.

The training cutoff means all models have frozen knowledge — a hard limit on what they can know about events, technologies, and conditions that emerged after their training data was collected. The context window means all models have finite working memory within a session, and that information that falls outside the window becomes invisible to them. The fluency-accuracy gap means that confident-sounding output is not reliable-output — the same tone of voice accompanies correct and incorrect statements.

These are not bugs. They are logical consequences of how these systems are built. Understanding them lets you predict where failures are likely, how to structure your work to avoid them, and how to evaluate output with appropriate skepticism.

The next chapter translates these mechanics into practical mental models — the conceptual frames that guide effective AI collaboration. The mechanics explain the behavior; the mental models tell you what to do about it.

Continue to Chapter 3: The Right Mental Models for AI Collaboration

Supporting materials: Exercises | Quiz | Case Study: The Frozen Clock Problem | Case Study: Elena and the Vanishing Memory | Key Takeaways | Further Reading