Chapter 29 — Case Study 2: The Dashboard That Looked Great While the Business Shrank

A labeled composite. This case is not a single real company; it is built from a pattern so common in the SEO industry that most practitioners have watched some version of it play out. Every number is a constructed teaching example. The pattern — vanity-metric reporting plus attribution overclaim, with no baseline and no connection to revenue — is entirely real; the specifics are illustrative.

Background

"Northlake Outfitters" (constructed) is a mid-size regional retailer with a growing website and an owner, Dana, who knows retail cold and SEO not at all. Eighteen months ago Dana hired an agency on a healthy monthly retainer to "grow our organic traffic." Every month since, a polished PDF has arrived: three big green charts, an upward trend on all of them, and a cheerful caption. Dana cannot read the charts closely — that is rather the point of paying an agency — but green arrows feel like progress, and the invoices keep getting paid.

The reporting failure

The monthly report was a museum of §29.2's vanity metrics, delivered with §29.5's attribution overclaim, on a foundation §29.1 says you must never skip and the agency did.

FIGURE CS29.2 — "Eighteen months of green"                     [constructed teaching example]
  WHAT THE REPORT SHOWED (every month, rising)
    · Keywords ranked ............ 4,900  → 14,200      (▲ "up 190%!")
    · Domain Authority ........... 21     → 31          (▲ "+10 points")
    · Total impressions .......... 240k   → 900k        (▲ "3.7x reach!")
    · Caption: "Outstanding growth — your SEO is thriving."
    · Occasionally: "SEO drove an estimated $180,000 in revenue." (last-click, unexplained)

  WHAT THE REPORT NEVER SHOWED
    · a baseline for anything that mattered (revenue, leads, non-branded traffic)
    · organic CLICKS or click-through rate (only impressions)
    · organic CONVERSIONS — purchases, the thing a retailer lives on
    · the BRANDED vs NON-BRANDED split (was the 'growth' just their own name?)
    · which pages or terms actually sold anything
    · a single month where ANYTHING went down

Underneath the green, three things were quietly true. The keyword count ballooned because the agency had published hundreds of thin blog posts that ranked on page four or five for long-tail phrases nobody searched — impressions with almost no clicks. The Domain Authority rise reflected some low-quality links and the vendor's own model changes, and meant nothing to Google. And the impressions grew largely because the site now appeared for a wide smear of irrelevant queries. Meanwhile the number nobody was reporting — organic revenue — had actually drifted down, as a core update devalued the thin content and the genuinely valuable category pages went unattended.

What it shows

This is the chapter's nightmare made concrete: a report that is entirely true and entirely useless. Not one figure in that PDF was fabricated. Every green arrow was real. And the whole document was a lie of emphasis, because it answered questions the business never asked (how many keywords? how high the DA?) and hid the only one that mattered (did we sell more?).

  • Vanity metrics misdirect (§29.2). Keyword count, DA, and raw impressions all rose while the business fell — the precise scenario the chapter warns about, in which the metrics on the report are uncorrelated with (or even inversely related to) the outcome the owner cares about.
  • No baseline meant no accountability (§29.1). Because the agency never froze a starting line for revenue or non-branded traffic or conversions, there was no number against which "up 190%" could be exposed as irrelevant. You cannot catch a misdirecting report if there is nothing real to compare it to.
  • Attribution was overclaimed (§29.5). "SEO drove \$180,000" was a last-click figure presented as established fact — no note that it was attributed, not caused, and no acknowledgment that organic revenue overall had declined. It was the exact "organic-caused" overreach the chapter tells you never to commit.
  • The absence of any miss was the tell (§29.7). Eighteen straight months without a single down number is not a triumph; it is a report engineered to have no bad news in it — the warning sign the chapter names outright.

Outcome

The reckoning came from outside the report, as it usually does. Dana's finance manager, doing an unrelated annual review, noticed that online revenue was flat-to-down over the same eighteen months the "SEO" charts had been soaring, and asked the obvious question: if SEO is up 190%, where is the money? The agency had no answer, because its report had never been about money. Dana churned within the quarter, trust destroyed — and, worse, now believed "SEO doesn't work," when in truth SEO reporting had failed her. A new consultant had to spend the first two months just rebuilding a real baseline, pruning the thin content (Chapter 12), splitting branded from non-branded traffic, wiring up purchase conversions in GA4, and — hardest of all — re-earning the credibility the green charts had spent.

The lesson

A rising chart is not a result, and a report with no bad news is not a good report. The vanity metrics did not deceive because they were false; they deceived because they were irrelevant, dressed as success and never checked against the business. The defense is everything this chapter teaches, applied without exception: freeze a real baseline; report the five KPIs that connect to revenue, not the three that photograph well; split branded from non-branded; say "attributed," never "caused"; and put the misses on the page next to the wins. The single question that would have saved this engagement is the one Dana's finance manager finally asked and the agency never did — what decision would this number change? Ask it of every chart before it reaches the owner, and a report like this one becomes impossible to send.

Discussion questions

  1. Every number in the report was true. Explain precisely how a report can be 100% accurate and still be dishonest. What word other than "false" describes what went wrong?
  2. Which single missing metric, added to the report, would have exposed the failure fastest — and why that one?
  3. The agency claimed "SEO drove \$180,000." Rewrite that claim as an honest practitioner would state it, and explain what makes your version more defensible.
  4. Contrast this case with Case Study 1 (the GA4 migration). One is a failure of interpretation on top of the tools; the other a disruption of the tools beneath the interpretation. What does holding them side by side teach about where reporting can break?
  5. The new consultant's hardest task was "re-earning credibility." Why is a stakeholder who has been burned by a dishonest report harder to report to honestly afterward — and what does that imply about the cost of a single green-washed month?