Chapter 34 Exercises: Measuring AI ROI

Section A: ROI Fundamentals

Exercise 34.1 — The Four Pillars of AI Value

For each of the following AI projects, identify the primary value pillar (direct revenue, cost reduction, risk reduction, or strategic optionality) and estimate how you would measure the value. If the project creates value across multiple pillars, identify the primary and secondary pillars.

a) A fraud detection model for an online payment processor that flags suspicious transactions in real time. b) A customer service chatbot that handles 40 percent of inbound queries without human intervention. c) A product recommendation engine for a streaming music platform. d) A predictive maintenance system for a fleet of 200 delivery trucks. e) Building a unified customer data platform that consolidates data from twelve legacy systems. f) An AI-powered drug discovery platform that screens molecular compounds 100x faster than traditional methods.

Exercise 34.2 — Why AI ROI Is Harder

A VP of Technology at a mid-size insurance company says: "We calculate ROI for every IT project using the same formula: (benefit - cost) / cost. Why should AI be any different?"

Write a two-paragraph response that explains three specific reasons why AI ROI measurement requires a different approach than traditional IT ROI. Use concrete examples from the insurance industry where possible.

Exercise 34.3 — The Politics of ROI

Read the following scenario and answer the questions below.

An AI team has built a demand forecasting model for a consumer goods company. The supply chain VP reports that the model has reduced forecast error by 22 percent. The CFO asks, "What is the dollar value of a 22 percent reduction in forecast error?" The AI team estimates $8 million in annual savings from reduced inventory carrying costs and fewer stockouts. The CFO's team recalculates using different assumptions and arrives at $3.2 million. The supply chain VP produces a third estimate of $11 million that includes "strategic value" and "competitive advantage."

a) Why are the three estimates so different? What assumptions likely differ? b) Which estimate would you present to the board? How would you present the range? c) What process would you recommend to prevent this kind of disagreement in future projects?

Section B: Cost Analysis and TCO

Exercise 34.4 — The Hidden Cost Audit

Tom's table in Section 34.3 lists common gaps between budgeted and actual AI costs. Using the following project description, create a realistic cost estimate that accounts for hidden costs.

Project: An NLP model that reads customer support tickets and automatically routes them to the correct department. The team estimates 4 months of development time with one data scientist ($200K annual salary), one ML engineer ($220K annual salary), and access to labeled data from 50,000 historical tickets.

Create two cost estimates: a) Naive estimate: Include only the costs explicitly mentioned. b) Realistic estimate: Add costs for data labeling, compute, infrastructure, deployment, stakeholder alignment, operations (year 1), and a 3-month schedule overrun. Justify each addition with a one-sentence rationale.

Exercise 34.5 — TCO Calculation

Using the TCO formula from Section 34.9, calculate the 5-year TCO for the following AI project. Then calculate the TCO multiplier.

a) What is the 5-year TCO? b) What is the TCO multiplier? c) If the project generates $500,000 in annual value, what is the 5-year NPV at a 10% discount rate? d) At what annual value does the project break even (NPV = 0) over 5 years?

Exercise 34.6 — Cost Category Classification

Classify each of the following costs into the correct category from the AI Cost Taxonomy (Section 34.3): data, compute, talent, infrastructure, organizational, opportunity, or risk.

a) $15,000 per month for AWS GPU instances used to train models b) $120,000 to hire a contractor to label 200,000 images c) 80 hours of the Head of Marketing's time spent defining model requirements d) A $2 million product launch delayed by 3 months because the ML team was allocated to another project e) $45,000 per year for a model monitoring platform (Weights & Biases) f) $500,000 legal settlement from a biased lending model g) $8,000 per month for a third-party data feed of competitor pricing

Section C: Value Measurement and Attribution

Exercise 34.7 — Designing an A/B Test

You are the AI product manager for an e-commerce company. Your team has built a recommendation engine for the checkout page ("Customers who bought this also bought..."). Design an A/B test to measure the engine's revenue impact.

Specify: a) The treatment and control groups b) The primary metric and at least two secondary metrics c) The minimum experiment duration and how you determined it d) At least two confounding factors you need to control for e) How you would translate the test results into an annual revenue estimate

Exercise 34.8 — Attribution Method Selection

For each of the following AI systems, recommend the most appropriate attribution method (A/B testing, before/after comparison, or modeling-based attribution) and explain why.

a) A pricing optimization model that adjusts prices for 50,000 SKUs daily b) An AI system that pre-fills insurance claim forms, reducing processing time c) A chatbot that handles 35 percent of customer service inquiries d) A demand forecasting model that determines inventory levels for 800 stores e) An AI-powered email marketing system that personalizes subject lines

Exercise 34.9 — The Efficiency Gains Debate

An AI system saves each of 200 customer service agents an average of 45 minutes per day. The average fully loaded cost per agent is $65,000 per year.

a) Calculate the total time saved per year (in hours). b) Calculate the theoretical dollar value of the time saved. c) Now consider: the agents are salaried employees, and none will be laid off. The company expects them to "handle more complex cases" with the saved time. Critique the ROI calculation. Under what conditions is the theoretical value a real value? Under what conditions is it misleading? d) Propose a measurement framework that captures the actual value of the freed-up time.

Section D: Strategic and Indirect Value

Exercise 34.10 — Option Value Estimation

Athena's unified customer data platform cost $4.2 million to build. Using the simplified option value framework from Section 34.5, estimate the option value under the following assumptions:

a) Three future AI projects are planned that will use the platform. Each project is estimated to save $800,000 in development costs compared to building data infrastructure from scratch. b) The probability that each project will actually be built is 70%, 50%, and 30% respectively. c) The projects are expected to start in years 1, 2, and 3. Use a 10% discount rate.

Calculate: What is the total option value of the platform based on these three projects alone?

Exercise 34.11 — Competitive Positioning Analysis

A retail bank has been investing in AI-powered credit risk assessment for three years. Their model has been trained on 8 million loan applications with outcome data. A new competitor is entering the market and plans to build a similar model.

a) Estimate the competitor's cost and timeline to replicate the bank's AI capability. Consider: data acquisition, labeling, model development, regulatory approval, and the learning curve. b) What is the competitive "lead time" the bank has? How would you quantify its strategic value? c) Under what circumstances could the competitor leapfrog the bank despite the data disadvantage?

Exercise 34.12 — The Indirect Benefits Audit

Review the following list of claimed "indirect benefits" from an AI program. For each, assess whether the benefit is (i) real and measurable, (ii) real but hard to measure, or (iii) speculative/wishful thinking. Justify your assessment.

a) "Our AI projects have built a culture of data-driven decision making." b) "The data cleaning we did for the churn model also improved our marketing analytics." c) "Our AI team's expertise makes us more attractive to top engineering talent." d) "Our recommendation engine creates switching costs for customers." e) "The AI initiative has improved cross-functional collaboration between IT and marketing." f) "Our AI platform reduces the time for future model deployment from 6 months to 6 weeks."

Section E: Kill Decisions and Portfolio Management

Exercise 34.13 — Kill or Continue?

For each of the following projects, decide whether to kill, continue, or pivot. State which kill criteria (if any) have been triggered and explain your reasoning.

Project A: An AI model to predict employee attrition. After 9 months and $400,000, the model achieves AUC of 0.58 (barely better than random). The HR VP remains enthusiastic and wants to try adding more data sources.

Project B: A computer vision system for quality inspection in manufacturing. After 12 months and $1.1 million, the model detects 92 percent of defects (exceeding the target of 85 percent). However, the plant manager has retired, and the new plant manager prefers the existing manual inspection process.

Project C: An AI-powered demand forecasting model for a restaurant chain. After 6 months and $300,000, the model shows 15 percent improvement over baseline. But the chain has just been acquired by a larger company with its own demand forecasting system.

Project D: A natural language search engine for an internal knowledge base. After 5 months and $200,000, the prototype is promising (users rate it 4.2/5 in usability testing) but the IT team estimates $800,000 in additional infrastructure costs for deployment.

Exercise 34.14 — Portfolio Design

You are the Chief AI Officer of a healthcare company with a $10 million annual AI budget. Design a portfolio of 8-10 AI projects using the portfolio matrix from Section 34.8 (Quick Wins, Strategic Bets, Moonshots, Experiments). For each project:

a) Name and describe the project in one sentence b) Classify it (Quick Win, Strategic Bet, Moonshot, or Experiment) c) Assign a budget allocation d) Estimate the expected value and probability of success e) Define one kill criterion

Verify that your portfolio meets the allocation guidelines from Section 34.8 and that the expected portfolio value is positive even if all moonshots fail.

Exercise 34.15 — Sunk Cost Scenario

A financial services firm has spent $3.2 million developing an AI model to predict market movements for its trading desk. The model has been in development for 18 months. Recent testing shows the model performs no better than a simple moving-average strategy. The lead data scientist argues that with $800,000 more investment and six additional months, the model could be significantly improved by incorporating alternative data sources.

a) Identify the sunk cost fallacy in this scenario. b) What information would you need to make a rational decision about whether to continue? c) Write a one-page memo to the CTO recommending a course of action. Include quantitative reasoning.

Section F: The AIROICalculator

Exercise 34.16 — Calculator Setup

Using the AIROICalculator class from Section 34.10, set up an ROI analysis for the following project:

Project: AI-powered dynamic pricing for an airline. Development cost: $1.2 million. Deployment: $400,000. Annual operations: $180,000. Retirement: $60,000. Expected annual revenue increase: $5.5 million (confidence: 0.70, ramp: 6 months). Expected cost savings from reduced manual pricing analyst time: $800,000 (confidence: 0.85, ramp: 3 months). Strategic value from competitive data accumulation: $600,000 (confidence: 0.40, ramp: 12 months).

Write the Python code to: a) Initialize the calculator and add all costs and value streams b) Print the executive summary c) Run sensitivity analysis on annual value and interpret the results d) Run Monte Carlo simulation with 30% value uncertainty and 15% cost uncertainty

Exercise 34.17 — Comparative Analysis

Using the AIROICalculator, compare the following two projects and recommend which to fund (the organization can only fund one).

Project Alpha: Fraud detection model. Development: $350,000. Deployment: $150,000. Annual ops: $70,000. Annual value: $2.1 million (risk reduction, confidence: 0.80). Time horizon: 5 years.

Project Beta: Customer segmentation model. Development: $500,000. Deployment: $200,000. Annual ops: $90,000. Annual value: $3.5 million (revenue, confidence: 0.55). Time horizon: 5 years.

a) Calculate NPV, IRR, and payback period for both projects. b) Run Monte Carlo simulations for both. Which project has higher probability of positive NPV? c) Which project would you recommend and why? Consider both the quantitative metrics and qualitative factors.

Exercise 34.18 — Sensitivity Deep Dive

Run the sensitivity analysis on Athena's churn prediction model (the example from Section 34.10) for all four variables. Then answer:

a) Which variable has the greatest impact on NPV? b) At what confidence-weighted annual value does the project NPV become negative? c) What would the annual operations cost need to be for the project to have an NPV of zero? d) Create a "worst realistic case" by setting all variables to their most unfavorable values simultaneously. Is the project still viable?

Section G: Communication and Benchmarking

Exercise 34.19 — Executive Presentation

You are presenting the ROI of your company's AI program to the board of directors. The program includes six projects with the following results:

a) Calculate the portfolio-level ROI for deployed projects only. b) Write a three-slide executive summary (describe the content of each slide) following the dashboard design principles from Section 34.11. c) Write a one-paragraph narrative for the board using one of the three narrative strategies (customer story, counterfactual, or competitive frame).

Exercise 34.20 — The "So What" Test

Rewrite each of the following sentences to pass the "so what" test from Section 34.11.

a) "Our NLP model processes customer reviews with 91% accuracy." b) "We reduced model training time by 60% through infrastructure optimization." c) "The AI team has grown from 3 people to 12 people over two years." d) "Our recommendation engine uses a hybrid collaborative-content-based filtering approach." e) "We achieved a 0.89 AUC on the fraud detection test set."

Exercise 34.21 — Benchmarking Assessment

Using the maturity-level framework from Section 34.12, assess the AI maturity level of the following organization and recommend what benchmarks they should target.

Organization: A regional grocery chain with 120 stores. They have deployed two AI projects (demand forecasting and a customer loyalty recommendation engine). Both were developed by a three-person data science team with help from a consulting firm. There is no formal AI governance process. The CEO is enthusiastic about AI but the COO is skeptical. ROI has been measured informally ("the forecasting model seems to have helped").

a) What maturity level is this organization? b) What benchmarks should they target for the next 12 months? c) What are the two most important investments they should make to advance to the next maturity level?

Section H: Integration and Synthesis

Exercise 34.22 — Full ROI Analysis

Choose an AI use case relevant to your industry or interest area. Perform a complete ROI analysis including:

a) Problem statement and value hypothesis (1 paragraph) b) Complete cost breakdown using the AI Cost Taxonomy (Section 34.3) c) Value estimation across all four pillars, with confidence levels d) AIROICalculator implementation (write the Python code) e) Sensitivity analysis on the three most uncertain assumptions f) Monte Carlo simulation with interpretation g) Kill criteria (at least one per category: technical, business, economic, strategic) h) One-page executive summary suitable for a CFO

This exercise integrates all concepts from Chapter 34 and connects to the capstone project in Chapter 39.

Exercise 34.23 — The Capstone Preview

Review the AI project lifecycle from Chapter 6. For each stage of the lifecycle, identify the specific ROI measurement activity that should occur at that stage. Create a table mapping lifecycle stages to ROI activities, deliverables, and decision points.

This mapping will be essential for your capstone AI transformation plan in Chapter 39.

These exercises progress from conceptual understanding (Section A) through quantitative analysis (Sections B-C), strategic reasoning (Sections D-E), hands-on tool use (Section F), communication skills (Section G), and synthesis (Section H). Complete at least one exercise from each section for a well-rounded understanding of AI ROI measurement.