Chapter 21 Quiz: AI-Powered Workflows

Chapter 21 Quiz: AI-Powered Workflows

Multiple Choice

Question 1. What is the primary problem that Retrieval-Augmented Generation (RAG) solves?

(a) Language models are too slow for real-time applications.
(b) Language models generate plausible but factually incorrect responses when they lack domain-specific knowledge.
(c) Language models cannot process documents longer than 1,000 words.
(d) Language models require fine-tuning for every new domain.

Question 2. In the RAG pipeline, what is the correct order of steps during the querying (online) phase?

(a) Generate response → Retrieve documents → Embed query → Return answer
(b) Embed query → Retrieve similar chunks → Augment prompt → Generate response
(c) Chunk documents → Embed chunks → Store in vector database → Search
(d) Retrieve documents → Chunk documents → Embed chunks → Generate response

Question 3. An embedding is best described as:

(a) A compressed version of a document that reduces its file size.
(b) A dense vector representation of text in a continuous space where semantically similar texts are mapped to nearby points.
(c) A keyword index that maps words to the documents containing them.
(d) A hash function that converts text to a unique identifier.

Question 4. Tom discovered that changing his chunking strategy improved retrieval quality by 25 percent, while switching embedding models improved it by only 3 percent. This finding suggests that:

(a) Embedding models are unimportant and should not be evaluated.
(b) Chunking strategy has a disproportionately large impact on RAG system quality relative to embedding model choice.
(c) More expensive embedding models always produce worse results.
(d) Fixed-size chunking is always superior to recursive chunking.

Question 5. What is the purpose of chunk overlap in a RAG pipeline?

(a) To increase the total number of chunks for better vector database performance.
(b) To ensure that information spanning a chunk boundary is captured in at least one complete chunk.
(c) To reduce the cost of embedding by reusing computed vectors.
(d) To improve the LLM's ability to generate longer responses.

Question 6. A customer asks: "Can I return SKU-4892A?" A semantic (dense) search retrieves chunks about general return policies. A keyword (sparse) search retrieves the specific product page for SKU-4892A. Which retrieval strategy would most likely find both relevant documents?

(a) Semantic search only
(b) Keyword search only
(c) Hybrid search (combining semantic and keyword search)
(d) Multi-query retrieval without hybrid search

Question 7. Which of the following best describes the difference between a chatbot and an AI agent?

(a) Chatbots use language models while agents use rule-based systems.
(b) Agents can plan and execute multi-step tasks using tools, while chatbots primarily generate text responses.
(c) Chatbots are faster than agents because they don't require external API calls.
(d) Agents always operate without human oversight, while chatbots always require human approval.

Question 8. In function calling, the LLM's role is to:

(a) Execute the function directly against the external system.
(b) Decide which function to call and generate the appropriate structured arguments, while the application executes the actual function.
(c) Generate the source code for the function from scratch.
(d) Replace the need for an API by simulating the function's behavior.

Question 9. Athena's RAG system retrieved an outdated price-match policy that incorrectly included Best Buy. This error illustrates that:

(a) RAG systems cannot handle multiple versions of the same document.
(b) Semantic search does not work for policy documents.
(c) RAG systems are only as accurate as the documents in their knowledge base, making data governance essential.
(d) LLMs always prefer older documents over newer ones.

Question 10. Which of the following is NOT a valid strategy for reducing RAG system latency?

(a) Caching responses for frequently asked queries.
(b) Using a smaller, faster LLM for routine queries.
(c) Streaming the LLM's response as it is generated.
(d) Increasing the number of chunks retrieved from 5 to 50.

Question 11. A RAG system generates an answer that includes a claim not found in any of the retrieved documents. Which evaluation metric would specifically detect this failure?

(a) Answer relevance
(b) Faithfulness
(c) Context precision
(d) Retrieval recall

Question 12. Cosine similarity is preferred over Euclidean distance for comparing text embeddings because:

(a) Cosine similarity is faster to compute.
(b) Cosine similarity normalizes for vector magnitude, so a longer document does not automatically appear more similar.
(c) Euclidean distance cannot be computed for high-dimensional vectors.
(d) Cosine similarity always produces values between 0 and 1.

Question 13. Which vector database would be most appropriate for a team that is already running PostgreSQL and wants to add vector search without introducing a new database technology?

(a) Pinecone
(b) ChromaDB
(c) pgvector
(d) Milvus

Question 14. An enterprise RAG system costs $0.50 per day in embedding and LLM generation costs, but saves customer service agents an estimated $40,000 per month in handle time. This example illustrates that:

(a) RAG systems are always inexpensive to operate.
(b) The technology cost of RAG is often trivial compared to the business value it creates.
(c) Customer service agents should be replaced entirely by RAG systems.
(d) LLM costs will continue to decrease indefinitely.

Question 15. Which of the following statements about AI agents is most accurate?

(a) AI agents should be given maximum autonomy to be most effective.
(b) Production AI agents typically operate with guardrails including predefined tool sets, confirmation steps for high-stakes actions, and human handoff capabilities.
(c) AI agents have fully resolved the accountability question for enterprise deployments.
(d) AI agents cannot be combined with RAG systems.

Short Answer

Question 16. Explain why hybrid search (combining semantic and keyword retrieval) is recommended for enterprise RAG systems. Provide one example of a query type where semantic search excels and one where keyword search excels.

Question 17. Describe three components of Athena's Knowledge Base Governance Process and explain how each component addresses the stale document problem.

Question 18. NK proposes extending Athena's internal policy co-pilot to a customer-facing chatbot. Ravi expresses caution. Describe two specific risks of a customer-facing RAG chatbot that do not apply to an internal agent-facing tool, and propose one design decision that mitigates each risk.

Scenario Analysis

Question 19. A mid-size insurance company wants to build a RAG system for its claims adjusters. The knowledge base includes 15,000 policy documents, state regulations for all 50 US states, and internal claims processing guidelines. Claims adjusters handle approximately 300 queries per day.

(a) Identify two metadata fields (beyond source document and date) that would be especially important for this knowledge base, and explain why.

(b) The company's general counsel insists that any AI-generated response to a claims adjuster must be verifiable against the source document. How does RAG's architecture support this requirement?

(c) An adjuster in California asks about coverage for flood damage. The system retrieves a document about flood coverage in Texas. What likely went wrong, and how would you fix it?

Question 20. Professor Okonkwo states: "RAG does not solve the problem of data quality. It concentrates it." In 150-200 words, explain what she means and discuss the implications for organizations that adopt RAG without a data governance strategy.

Answer key: 1-b, 2-b, 3-b, 4-b, 5-b, 6-c, 7-b, 8-b, 9-c, 10-d, 11-b, 12-b, 13-c, 14-b, 15-b. Short answer and scenario responses are evaluated by the instructor. Sample responses are provided in Appendix B.