Chapter 31 Quiz: Marketing Analytics and Campaign Analysis

Instructions: Select the best answer for each multiple-choice question. For open-response questions, write your answer in complete sentences. The answer key follows all questions.

Multiple Choice (Questions 1–12)

Question 1

A company spent $120,000 on marketing last quarter and acquired 300 new customers. What is the Customer Acquisition Cost?

A) $40 B) $400 C) $4,000 D) $40,000

Question 2

A SaaS company has a CAC of $800 and an LTV of $3,200. Which statement best describes the health of this business's acquisition economics?

A) The business is unprofitable — LTV is too low to justify the CAC B) The ratio is 4:1, which is generally considered healthy for a scaling business C) The ratio is 4:1, which is dangerously close to the minimum acceptable threshold D) The business should immediately reduce CAC by cutting marketing spend

Question 3

Your email campaign had 28,000 recipients, 7,280 opens, and 1,092 clicks on the main link. What is the click-through rate?

A) 3.9% B) 15.0% C) 26.0% D) 7.3%

Question 4

Last-touch attribution would be MOST useful for answering which question?

A) How do customers first discover our brand? B) Which channel does the most work in the middle of the customer journey? C) What is the final interaction that closes deals? D) Which channels should get the most total credit for revenue?

Question 5

In an A/B test, you obtain a p-value of 0.032. The significance threshold is 0.05. What is the correct interpretation?

A) There is a 3.2% probability that the treatment is better than the control B) There is a 96.8% probability that the treatment will improve performance in the future C) If there were truly no difference, there is a 3.2% probability of observing a result this extreme by chance D) The test is inconclusive — you need to collect more data

Question 6

Which of the following represents the most dangerous A/B testing mistake in terms of inflating false positive rates?

A) Running the test for exactly the planned duration B) Checking results daily and stopping when the result first appears significant C) Using a significance threshold of 0.05 instead of 0.01 D) Calculating sample size before running the test

Question 7

A company has a 35% gross margin on its products. What is the minimum ROAS it needs to achieve to break even on ad spend?

A) 2.86 B) 3.50 C) 0.35 D) 1.35

Question 8

In a marketing funnel with stages: Visit → View Product → Add to Cart → Checkout → Purchase, you observe these counts: 10,000 → 4,500 → 1,800 → 810 → 405. Which transition has the largest absolute drop-off?

A) Visit → View Product B) View Product → Add to Cart C) Add to Cart → Checkout D) Checkout → Purchase

Question 9

You run 20 simultaneous A/B tests using p < 0.05 as your threshold. How many false positives should you expect by chance, assuming all null hypotheses are true?

A) 0 B) 1 C) 5 D) 10

Question 10

The UTM parameter utm_medium in the URL ?utm_source=google&utm_medium=cpc&utm_campaign=fall_promo indicates:

A) The specific campaign that sent the traffic B) The type of marketing channel (paid search in this case) C) The specific keyword that triggered the ad D) The name of the piece of content

Question 11

A consulting firm's referral channel has a CAC of $200 and an average client LTV of $24,000. Their LinkedIn advertising has a CAC of $1,800 and an average client LTV of $8,400. Based solely on LTV:CAC ratio, which channel is more efficient, and by approximately how much?

A) Referrals — 120× vs 4.7×, making referrals approximately 25× more efficient B) LinkedIn — lower CAC is always more efficient C) Both are equally efficient because LTV is higher for referrals D) Cannot determine without knowing total spend on each channel

Question 12

A campaign cohort analysis shows that customers acquired through the "Holiday Preview" campaign had the highest Month 0 revenue but the lowest Month 5 revenue among four cohorts. What does this most likely indicate?

A) The Holiday Preview campaign was the best-performing campaign overall B) Holiday Preview customers were likely one-time seasonal buyers with low retention C) The analysis has an error — a campaign cannot have high Month 0 and low Month 5 revenue D) The company should run Holiday Preview campaigns more frequently

Short Answer (Questions 13–17)

Question 13

Explain the difference between statistical significance and business significance. Give a concrete example of a result that might be statistically significant but not worth acting on.

(4–6 sentences)

Question 14

A marketing manager argues: "Our paid social campaigns have lower ROAS than our email campaigns, so we should cut social and put everything into email." Write a two-paragraph response explaining at least two reasons why this logic might be flawed.

Question 15

Describe what a "novelty effect" is in the context of A/B testing, explain why it matters, and suggest how you would design a test to account for it.

(3–5 sentences)

Question 16

Define the CAC payback period. A company has a CAC of $600, earns $75/month per customer, and has a 55% gross margin. Calculate the payback period. If this company is in SaaS, is this result acceptable? Explain.

Question 17

You are looking at a marketing funnel where 10,000 visitors reach the homepage but only 82 make a purchase (0.82% overall conversion). A colleague says: "Our conversion rate is terrible — we need to redesign the entire website." How would you use funnel analysis to give a more nuanced and useful answer?

(4–6 sentences)

Answer Key

Multiple Choice Answers

Q1: B — $400 CAC = $120,000 / 300 = $400. Option A misplaces a decimal; C and D move it further. Always sanity-check CAC calculations by confirming the units make sense in context.

Q2: B — The ratio is 4:1, which is generally considered healthy for a scaling business An LTV:CAC of 4:1 ($3,200 / $800) falls in the 3:1–5:1 "healthy" range. Option C is incorrect — 4:1 is well above the danger zone. Option D is a common misapplication of this metric; a healthy ratio means you can profitably grow, not that you should cut spend.

Q3: A — 3.9% CTR = Clicks / Recipients = 1,092 / 28,000 = 3.9%. Option B (15.0%) is the click-to-open rate (1,092 / 7,280), a different metric. Option C is the open rate. Always clarify which denominator you are using when discussing CTR for email.

Q4: C — What is the final interaction that closes deals? Last-touch attribution credits the final touchpoint before conversion, making it best suited for understanding what "closes" customers. First-touch (option A) is better for discovery questions. Neither model is good for option B or D, which require multi-touch approaches.

Q5: C — If there were truly no difference, there is a 3.2% probability of observing a result this extreme by chance This is the precise definition of a p-value. Option A confuses p-value with the probability that the specific treatment is better. Option B is wrong — p-values say nothing about future performance. Option D is wrong — p = 0.032 < 0.05 means the result IS significant.

Q6: B — Checking results daily and stopping when the result first appears significant This is "peeking," which exploits natural random fluctuations in early data to produce false positives at a far higher rate than the nominal alpha suggests. Running tests for the planned duration (A), using a conservative threshold (C in reverse), and pre-calculating sample size (D) are all good practices.

Q7: A — 2.86 Break-even ROAS = 1 / Gross Margin = 1 / 0.35 = 2.857. At this ROAS, every dollar of ad spend generates exactly enough gross profit to cover the ad cost. A ROAS of 3.50 (option B) exceeds break-even but is not the calculation result.

Q8: A — Visit → View Product Absolute drop-off: Visit → View Product = 5,500; View Product → Add to Cart = 2,700; Add to Cart → Checkout = 990; Checkout → Purchase = 405. The first transition loses the most visitors in absolute terms, even though the drop-off rate is similar at other stages.

Q9: B — 1 Expected false positives = n_tests × alpha = 20 × 0.05 = 1. This is the whole point of multiple testing correction: running many tests simultaneously inflates the family-wise error rate even when the per-test alpha is controlled.

Q10: B — The type of marketing channel (paid search in this case) utm_medium indicates the marketing medium (cpc, email, organic, social, referral). utm_source identifies where the traffic came from (google, facebook, etc.). utm_campaign names the campaign, and utm_term tracks keywords.

Q11: A — Referrals — 120× vs 4.7×, making referrals approximately 25× more efficient Referral LTV:CAC = $24,000 / $200 = 120. LinkedIn LTV:CAC = $8,400 / $1,800 = 4.67. The referral channel is approximately 26× more efficient by this metric. Option B is wrong — CAC alone does not determine efficiency without considering LTV.

Q12: B — Holiday Preview customers were likely one-time seasonal buyers with low retention This pattern — high initial spend, rapid drop-off — is characteristic of promotional or event-driven campaigns that attract price-sensitive or seasonal buyers rather than building a loyal customer base. The month-by-month cohort data tells the retention story that overall revenue numbers hide.

Short Answer Guidance

Q13 (Sample Answer) Statistical significance means the probability of observing our result by random chance (if there were no true effect) is below our threshold. Business significance means the effect is large enough to actually matter for business decisions. A test could be statistically significant but not business significant when the effect size is real but trivially small.

Example: An A/B test on a checkout page finds a statistically significant increase in conversion rate from 8.000% to 8.002%, with p = 0.02. With millions of visitors, even tiny effects become detectable. But a 0.002 percentage point improvement might translate to 50 additional sales per year — far too small to justify the development time. Statistical machinery does not know what "important" means; you have to supply that judgment.

Q14 (Sample Answer) First paragraph: The comparison is measuring different things. Email campaigns typically reach existing customers or warm prospects who already know the brand. Paid social often reaches entirely new audiences. A lower ROAS on social may simply reflect the fact that new customer acquisition is inherently more expensive than re-engaging existing relationships — not that social is failing. You would need to compare the type of conversion (new customer vs. returning) before drawing any conclusions.

Second paragraph: Diminishing returns mean you cannot simply triple your email budget and get triple the email revenue. Email lists have finite sizes, and beyond a certain frequency, you hurt deliverability and unsubscribe rates. You need both acquisition channels (like social) to build the audience and retention channels (like email) to monetize it. Cutting social entirely would eventually starve your email list of new entrants, which would cause email revenue to decline over time.

Q15 (Sample Answer) A novelty effect occurs when a new version of something performs better in a test not because it is genuinely better, but simply because it is unfamiliar and therefore catches users' attention. After the novelty wears off, performance reverts toward baseline. It matters because it can lead you to "launch" a winning variant that stops winning after a week.

To account for it, run tests long enough to outlast the novelty window — typically at least 2–3 full business cycles (often 2 weeks minimum). For features with high visibility, consider adding a "freshness check" by segmenting the data into early-test and late-test periods and verifying that the treatment's advantage holds in both.

Q16 (Calculation and Answer) CAC Payback Period = CAC / (Monthly Revenue per Customer × Gross Margin %) = $600 / ($75 × 0.55) = $600 / $41.25 = 14.5 months

For SaaS, the typical benchmark is under 12 months. At 14.5 months, this company is slightly above the target — it is not alarming, but it means the company is funding customer acquisition for over a year before recouping the investment. If the company is growing quickly, the cash flow implications compound, as each new customer cohort must be funded before it pays back. The company should monitor whether LTV is high enough to justify the slower payback, or whether CAC can be reduced through channel optimization.

Q17 (Sample Answer) Rather than redesigning the entire website based on overall conversion rate, a funnel analysis would show where specifically the most visitors are dropping off. Perhaps 90% of the conversion problem is concentrated in two transitions rather than spread evenly. For example, if "Add to Cart → Checkout" shows a 65% drop-off rate while other transitions are near benchmark, the problem is abandonment behavior at checkout — not the homepage or product pages. Redesigning the whole site to fix a checkout problem would waste significant resources. The funnel isolates the problem to specific pages and user actions, allowing targeted investment where it will have the highest return.