Quiz: Large Language Models — How ChatGPT and Its Peers Work

Test your understanding of key concepts from Chapter 5. Answers appear at the end.

Question 1. What is the core mechanism by which large language models generate text?

A) They search a database for the most relevant pre-written answer B) They predict the most likely next token given all preceding tokens C) They understand the question and reason through to a logical answer D) They match keywords in the prompt to stored responses

Question 2. A "token" in the context of LLMs is best described as:

A) A unit of digital currency used to pay for AI access B) A complete word in the English language C) A chunk of text (word, word fragment, or punctuation mark) that the model processes D) A security credential that authenticates the user

Question 3. What does the "temperature" setting control in an LLM?

A) How fast the model generates text B) The level of randomness in token selection C) The maximum length of the response D) How polite the model's responses are

Question 4. The transformer architecture's key innovation is the "attention" mechanism. Which of the following best describes what it does?

A) It ensures the model pays attention to the user's question B) It allows every token in a sequence to weigh the relevance of every other token C) It filters out irrelevant information from the training data D) It measures how long the user focuses on each part of the response

Question 5. Pre-training an LLM involves:

A) A small team of experts manually writing all the model's knowledge B) Training the model on a massive dataset to learn statistical patterns of language C) Programming specific rules for grammar and factual accuracy D) Connecting the model to a live internet feed so it always has current information

Question 6. Why might an LLM that was pre-trained but not fine-tuned behave strangely when asked a direct question?

A) Pre-training teaches the model to predict text continuations, not to answer questions helpfully B) Pre-training only covers scientific topics, not conversational ones C) The model hasn't been connected to the internet yet D) Pre-training intentionally makes models unhelpful to prevent misuse

Question 7. RLHF (Reinforcement Learning from Human Feedback) primarily teaches the model to:

A) Search the internet for accurate answers B) Produce responses that human evaluators rate as helpful, harmless, and honest C) Understand the true meaning of human language D) Correct all factual errors in its training data

Question 8. When an LLM generates a confident, well-formatted citation that refers to a paper and author that don't exist, this is called:

A) A system error B) A bug in the code C) A hallucination D) Intentional deception

Question 9. The "knowledge cutoff" of an LLM refers to:

A) The maximum number of facts the model can store B) The date after which the model has no training data and cannot reliably report events C) The point where the model stops learning during a conversation D) A safety feature that prevents the model from sharing sensitive information

Question 10. The "stochastic parrot" characterization of LLMs argues that:

A) LLMs are becoming truly intelligent and should be given rights B) LLMs produce language without genuine understanding, recombining patterns with randomness C) LLMs are useless for practical purposes D) LLMs will replace all human communication within a decade

Question 11. Which of the following is a task that current LLMs do well?

A) Verifying the factual accuracy of their own outputs B) Generating coherent, well-structured text across many genres C) Performing reliable multi-step mathematical calculations D) Accessing real-time information about current events

Question 12. Which of the following is a task that current LLMs systematically struggle with?

A) Translating between common language pairs B) Adapting tone and style based on instructions C) Maintaining perfect factual accuracy without external verification D) Generating computer code in multiple programming languages

Question 13. In Priya's story, she discovered that the chatbot generated a fake citation. This happened because:

A) The model was programmed to create fake citations to test students B) A bug in the software corrupted a real citation C) The model predicted that a plausible-sounding citation was the most likely text to follow, without any mechanism to verify its accuracy D) The model's internet connection was interrupted during generation

Question 14. The chapter argues that the human workers who perform RLHF annotation:

A) Are highly paid executives at major tech companies B) Have no influence on the model's behavior C) Often work as contractors under difficult conditions, sometimes reviewing disturbing content D) Are entirely automated — no real humans are involved

Question 15. Constitutional AI (CAI) differs from RLHF in that:

A) It uses no training at all B) It gives the model written principles to evaluate its own outputs, rather than relying entirely on human rankings C) It makes the model perfectly safe D) It eliminates the need for any pre-training

Question 16. The chapter's threshold concept — "LLMs predict the next word; they don't understand meaning" — has which practical implication?

A) LLM outputs are always wrong and should never be used B) LLM outputs require human verification because accuracy is incidental to the generation process, not guaranteed by it C) LLMs will never be useful for any professional task D) Only programmers should be allowed to use LLMs

Question 17. Why does the concentration of frontier LLM development among a few organizations matter?

A) It makes the models slower to use B) It means only those organizations can decide training data, model behavior, and safety standards C) It guarantees higher quality because fewer companies means more focus D) It has no significant implications for society

Question 18. Priya's revised approach to using LLMs is best summarized as:

A) Never use them — they're too unreliable B) Trust them completely — they're usually right C) Use them as brainstorming tools while independently verifying all specific claims D) Only use them for creative writing, never for academic work

Answer Key

  1. B — LLMs generate text by predicting the most likely next token. They don't search databases, and the process involves statistical prediction rather than understanding.

  2. C — Tokens are chunks of text that may be whole words, word fragments, or punctuation — the basic unit the model works with.

  3. B — Temperature controls how willing the model is to select lower-probability tokens. Low temperature = predictable; high temperature = more random/creative.

  4. B — Self-attention allows every token to assess the relevance of every other token in the sequence, enabling the model to capture relationships across the full input.

  5. B — Pre-training involves learning language patterns from massive datasets through next-token prediction across trillions of examples.

  6. A — Pre-training teaches text prediction/continuation, not question-answering behavior. A raw pre-trained model might continue a question with more questions rather than providing an answer.

  7. B — RLHF uses human rankings to teach the model to produce outputs that align with human preferences for helpfulness, harmlessness, and honesty.

  8. C — Hallucination is when an LLM generates confident, fluent text that is factually incorrect or entirely fabricated. It's not a bug or deception — it's a structural feature of next-token prediction.

  9. B — The knowledge cutoff is the date beyond which the model lacks training data, meaning it cannot reliably report on events after that date.

  10. B — The stochastic parrot argument holds that LLMs produce language through pattern recombination with randomness, without genuine understanding of meaning.

  11. B — LLMs excel at generating coherent, well-structured text across genres. They do not reliably verify facts, perform exact math, or access real-time information without special tooling.

  12. C — Factual accuracy without external verification is a systematic weakness of LLMs due to the gap between pattern prediction and truth.

  13. C — The model predicted plausible-looking citation text (name, institution, year, page number) because that pattern frequently appeared in its training data. It had no mechanism to verify the citation's accuracy.

  14. C — RLHF annotation work is often performed by contractors, sometimes in lower-income countries, who may face low pay, limited support, and exposure to disturbing content.

  15. B — CAI provides the model with explicit written principles for self-evaluation, reducing (but not eliminating) reliance on human ranking.

  16. B — Since accuracy is a byproduct of pattern matching rather than a guaranteed property, human verification of LLM outputs is essential.

  17. B — Concentration means a small number of organizations make consequential decisions about what models learn, how they behave, and what safety measures are implemented.

  18. C — Priya's approach treats the LLM as a brainstorming partner while maintaining responsibility for verifying all factual claims independently.