Chapter 33 Quiz: AI Product Management

Multiple Choice

Question 1. NK's loyalty personalization engine achieves 78% relevance. The current non-personalized homepage has approximately 12% relevance. Which framing most accurately communicates the AI product's value to an executive audience?

a) "The AI is wrong 22% of the time." b) "The AI achieves 78% relevance, exceeding the industry benchmark." c) "The AI is 6x more relevant than the current experience and generates an estimated $2.4M in incremental annual revenue." d) "The AI's F1 score is 0.78, which is statistically significant at p < 0.01."

Question 2. What is the primary reason that AI product management is more challenging than traditional product management?

a) AI products require more expensive engineering talent b) AI products are probabilistic, requiring the PM to manage uncertainty in performance, set non-binary acceptance criteria, and design for graceful failure c) AI products take longer to build d) AI products have more competitors

Question 3. Which of the following is NOT one of the five user mental models of AI described in this chapter?

a) The "magic" model — AI knows everything and never makes mistakes b) The "database" model — AI looks up answers in a giant database c) The "calculator" model — AI performs rapid mathematical computations d) The "surveillance" model — AI works by collecting extensive personal data

Question 4. In NK's A/B/C test of recommendation display designs at Athena, Design C ("Your Picks + Why" with an expandable explanation and a "Not for me" button) outperformed the other designs. What was the most important benefit of Design C beyond higher click-through rates?

a) It was cheaper to implement b) It generated 40% more user feedback, which fed back into the model and improved future recommendations c) It required less model accuracy to achieve the same user satisfaction d) It reduced the need for A/B testing in future iterations

Question 5. A company has built an AI-powered customer service chatbot. The PM defines the following degradation levels: (1) full AI response, (2) AI response with reduced confidence disclaimer, (3) template-based responses from a rules engine, (4) connection to a human agent, (5) "We're experiencing high volume — please try again later" message. Which principle does this hierarchy implement?

a) Probabilistic scaling b) Graceful degradation c) Model retraining d) Continuous deployment

Question 6. According to this chapter, what is the "perfection trap" in AI product management?

a) Shipping an AI product that is too accurate, making users overly dependent on it b) Delaying the launch of an AI product until it reaches near-perfect performance, even when it is already better than the existing alternative c) Setting performance thresholds so low that the AI product ships before it is useful d) Assuming that a perfect model will automatically create a perfect product

Question 7. Which MVP approach involves humans performing the AI's job behind the scenes while the product appears AI-powered to users?

a) Rules-based MVP b) Transfer learning MVP c) Wizard of Oz MVP d) Data-first MVP

Question 8. An AI PM discovers that the recommendation engine is creating a "filter bubble" — recommending an increasingly narrow set of products as users interact with the system. This is an example of:

a) A virtuous feedback loop b) A vicious feedback loop c) Concept drift d) An infrastructure failure

Question 9. Professor Okonkwo's roadmap heuristic suggests that in the early stage (0-6 months post-launch), the investment split should be approximately:

a) 70% model improvement, 20% feature development, 10% infrastructure b) 40% model improvement, 30% feature development, 30% infrastructure c) 25% model improvement, 50% feature development, 25% infrastructure d) Equal thirds: 33% model improvement, 33% feature development, 33% infrastructure

Question 10. When NK presents the personalization engine results to Athena's CEO, she voluntarily discloses three known weaknesses. Why is this communication strategy effective?

a) It demonstrates that NK has low expectations for the product b) It prevents the CEO from discovering the weaknesses independently, which would erode trust; volunteering limitations builds credibility c) It signals to the engineering team that more resources are needed d) It is required by Athena's AI governance framework

Question 11. Which of the following best describes the difference between the "feature track" and "performance track" of an AI product roadmap?

a) The feature track covers front-end development; the performance track covers back-end development b) The feature track shows deliverables with dates (commitments); the performance track shows model accuracy targets with uncertainty ranges (goals) c) The feature track is for the engineering team; the performance track is for the data science team d) The feature track measures user engagement; the performance track measures business revenue

Question 12. An AI PM is evaluating an A/B test for a new recommendation feature. The test has been running for three days and shows a 45% increase in click-through rate for the treatment group. What should the PM do?

a) Launch the feature immediately — 45% lift is exceptional b) Continue running the test for at least 2-4 weeks to wash out novelty effects and capture true behavioral change c) Increase the treatment group to 100% of users and stop the test d) Reduce the treatment group to 1% of users to minimize risk

Question 13. Which metric category is described as a "leading indicator of long-term engagement" for AI products?

a) Engagement metrics b) Quality metrics c) Trust metrics d) Business outcome metrics

Question 14. NK's A/B test revealed a +181% click-through rate lift but only a +30% purchase conversion lift. What does this discrepancy most likely indicate?

a) The AI model is performing poorly b) The AI is effective at generating interest, but the post-click experience (product pages, pricing, checkout) is the conversion bottleneck c) The A/B test was run incorrectly d) Users are clicking on recommendations out of curiosity but have no intention of purchasing

Question 15. An AI PM is writing acceptance criteria for a loan approval recommendation model. Which of the following is a fairness acceptance criterion?

a) "The model must process applications within 200 milliseconds" b) "Model approval rates must not differ by more than 5 percentage points across racial and ethnic groups after controlling for creditworthiness factors" c) "The model must achieve at least 85% accuracy on the test set" d) "The system must have 99.9% uptime"

Short Answer

Question 16. Explain the difference between a "model failure" and a "concept drift failure" in an AI product. Give one example of each for a product recommendation engine.

Question 17. A data scientist on your team says, "We improved the model's F1 score from 0.76 to 0.84. The product is now significantly better." As the AI PM, why might this statement be insufficient, and what additional information would you need before telling stakeholders the product has improved?

Question 18. Describe two specific strategies for addressing the "cold-start problem" in an AI-powered recommendation engine for new users who have no purchase history.

Question 19. Why does the chapter argue that AI MVPs are typically more expensive and time-consuming than traditional MVPs? Identify at least three components of an AI MVP that a traditional MVP does not require.

Question 20. Professor Okonkwo calls the AI PM a "trilingual translator." Name the three "languages" and give one example of translating the same piece of information into each language.

Scenario-Based

Question 21. You are the AI PM for a travel booking platform. Your AI-powered dynamic pricing model adjusts hotel prices in real-time based on demand, seasonality, competitive pricing, and user behavior. A customer complains on social media: "The price changed between when I searched and when I tried to book! This is a scam!" Your VP of Customer Experience asks you to address this.

a) Is this a model failure, a feature design issue, or expected behavior? Explain. b) What product design changes would you recommend to prevent similar complaints? c) Draft a response to the VP that explains the situation in non-technical language.

Question 22. You are the AI PM for a content moderation system at a social media company. The system automatically removes content that violates community guidelines. The model has 96% precision (4% of removed content was actually not in violation) and 88% recall (12% of violating content is not caught).

a) A civil liberties organization criticizes your platform for "algorithmic censorship" based on the 4% false positive rate. How would you respond? b) A parent organization criticizes your platform for "failing to protect children" based on the 12% miss rate. How would you respond? c) These two criticisms pull in opposite directions. As the AI PM, how do you balance them? What additional product features could help?

Question 23. Your company's CEO returns from a conference and says: "I just saw a demo of GPT-5. We need to integrate generative AI into every product by Q3." As the AI PM, write a response memo (5-7 bullet points) that is respectful of the CEO's enthusiasm while redirecting toward a strategic approach.

Answers to selected questions are provided in the appendix.