Case Study 2 — The Store That Tried to Block Its Way Out of Index Bloat (and Made It Worse)

A complementary angle. Case Study 1 showed the pipeline's controls being clarified by Google in public. This one shows them being misused by a competent team acting on reasonable-sounding advice — a failure case that teaches the exact limits of robots.txt, the reality of crawl budget at scale, and the right way to manage index bloat.

This is a labeled composite. It is not one named company; it is a pattern that has played out at countless e-commerce sites and that Google's Search Relations team has warned about for years. Every specific number below is illustrative — the mechanism, and how ordinary the mistake is, are what is real. (The deep treatment of faceted navigation is Chapter 31; of crawl budget at scale, Chapter 33.)

Background: a real catalog with a URL explosion

Picture a mid-size online retailer — call it a home-and-garden store with about 4,000 genuine product and category pages. Solid catalog, real inventory, decent content. But the site has faceted navigation: on every category page, shoppers can filter by color, size, brand, price range, material, and rating, and each filter selection changes the URL by adding a parameter. Multiply six filters with several values each, in any combination and any order, and the site quietly generates not 4,000 URLs but hundreds of thousands of distinct parameter URLs — ?color=blue, ?color=blue&size=large, ?size=large&color=blue, ?color=blue&size=large&sort=price, and on and on. Almost all of them show near-duplicate content: the same products, sliced and reordered.

The symptoms arrived in the usual order. A site: search that should have returned ~4,000 results returned something closer to 280,000. New products took a long time to get crawled and indexed. Search Console's Crawl stats showed Googlebot spending the overwhelming majority of its crawl requests on parameter URLs that generated no traffic, while real product pages waited. This is genuine index bloat, and — because the site is large and auto-generating URLs — it is one of the real cases where crawl budget actually matters (the exception §14.6 is careful to name).

So far, an accurate diagnosis. The failure was in the fix.

The issue: the plausible wrong move

A consultant recommended what sounds obviously right: block the parameter URLs in robots.txt so Google stops crawling them. The team added rules like:

User-agent: *
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?sort=

The logic seemed airtight: "these junk URLs are wasting our crawl budget and bloating our index, so let's stop Google from crawling them." The team shipped it and waited for the index to shrink.

It didn't. Here is what actually happened, and it is the chapter's marquee myth playing out at scale.

text FIGURE 14.5 — "Blocking the crawl didn't remove the pages" [constructed teaching example] BEFORE (the bloat) ~280,000 parameter URLs indexed; real products crawled slowly. THE MOVE Disallow the parameter patterns in robots.txt (block the crawl). WEEK 2 Index count barely moves. Many parameter URLs now show, in Search Console, "Indexed, though blocked by robots.txt." WHY The parameter URLs were linked — from the site's own faceted menus and from a few external pages. Google already KNEW the URLs. Blocking the crawl only stopped Google from RE-READING them; it did not remove them from the index, and it removed Google's ability to see any signal (including a noindex) on them. (All figures illustrative — the SHAPE, a block that fails to deindex, is the real lesson.)

Read that against §14.1. robots.txt controls crawling, not indexing. The parameter URLs were already discovered (the site linked to them from its own filter menus). Blocking the crawl did not delete them from the index — it merely froze them there as bare, snippet-less listings Google could no longer look inside, and it destroyed the team's ability to ever place a noindex on them, because Google could no longer crawl the pages to read one. The team had, with the best intentions, walked straight into the trap the whole chapter warns about: you cannot de-index by blocking the crawl. Worse, they'd made the bloat harder to fix than before they touched it.

What it shows

  1. Crawl budget and index bloat are real — at scale. This is the honest exception to "most sites don't need to care." A 4,000-page catalog behaving like a 280,000-page site has a genuine crawl-and-index problem. The chapter's "don't worry about crawl budget" advice is for the small and mid-size majority; this store is precisely the minority for whom it matters.

  2. The plausible fix was the wrong stage. The team correctly diagnosed bloat (an indexing problem) and then reached for a crawling tool. Diagnosis and treatment landed at different stages of the pipeline. The result was the classic robots.txt backfire: "Indexed, though blocked by robots.txt."

  3. Blocking the crawl removes your ability to fix the page. This is the subtle, cruel part. A crawlable junk URL can be noindexed, canonicalized, or redirected — you have options. A crawl-blocked junk URL that's already indexed is stuck: Google can't see any instruction you add. Blocking should be the last step, applied only after the pages are already out of the index, never the first.

Outcome: the correct sequence

The recovery required undoing the block and doing it in the right order — the sequence every technical SEO should be able to recite for faceted navigation:

  1. Un-block the parameter URLs in robots.txt so Google can crawl them again and see instructions.
  2. Decide, per facet, what each URL should be. Filters that create genuine, valuable landing pages (say, a "blue outdoor rugs" page people actually search for) get kept and canonicalized to themselves. The vast majority — arbitrary combinations and sort orders — get a noindex (so Google drops them from the index) and/or a canonical pointing at the clean category page (so signals consolidate).
  3. Wait for Google to re-crawl and honor the noindex, watching the index count fall in the Page indexing report over the following weeks.
  4. Only then, once the junk is out of the index, optionally add robots.txt blocks (or handle parameters at the architecture level) to save crawl budget going forward — because now there is nothing left to accidentally freeze in place.

Over roughly two months, the indexed count in the illustrative version falls from ~280,000 toward the real ~4,000, crawl requests re-focus on genuine products, and new products get indexed faster. The fix was never a single toggle; it was a sequence, in the right order, respecting which stage each tool operates on.

The lesson

Block the crawl last, not first — and never as a way to de-index. The store's team was competent and their diagnosis was right; they lost weeks because they applied a crawling tool to an indexing problem, and in doing so they removed their own ability to fix it. The transferable rule: to remove pages from the index, keep them crawlable and tell Google noindex (or 404/410, or canonicalize); reserve robots.txt blocking for genuine crawl-efficiency after the index is clean. And know the honest exception to the chapter's crawl-budget skepticism: at real scale, with real URL explosions, this stuff matters — which is exactly why doing it in the right order matters even more.

Connect it to Rivertown. Rivertown is nowhere near this scale (~120 real pages), so its own index bloat — the empty tag archives from §14's audit — is fixed the simple way: noindex the archives in its SEO plugin, keep them crawlable, and let Google drop them. The store's cautionary tale is what Rivertown would face if it grew into a large service×city matrix without discipline — the exact scenario we design against in Chapter 33.

Discussion questions

  1. The team's diagnosis (index bloat, wasted crawl) was correct, but their fix failed. Explain, using the crawl-versus-index distinction, why a correct diagnosis led to the wrong tool — and what one sentence of advice would have prevented it.
  2. "Indexed, though blocked by robots.txt" is the status that revealed the failure. In your own words, explain how a URL can be indexed and blocked at the same time.
  3. Reorder these four actions into the correct sequence and justify it: (a) add robots.txt blocks, (b) noindex the junk URLs, (c) un-block the URLs so they can be crawled, (d) wait for re-crawl and watch the index shrink.
  4. Some filter URLs (like "blue outdoor rugs") are worth keeping and ranking; most are not. What criterion would you use to decide which faceted URLs deserve to be indexable? (Chapter 31 is the full answer — reason it out first.)
  5. This case is one of the real exceptions to "most sites don't need to worry about crawl budget." How would you explain to a small client why their 300-page site doesn't have this problem, without dismissing the real problem this large store had?