Case Study 2 — The Catalog That Crawled Itself to Death (and the Fix That Made It Worse)

A complementary angle. Case Study 1 showed thin content at scale catching up with a giant from the outside — a quality update. This one shows a mid-size store wounding itself from the inside, twice: first by letting faceted navigation explode, then by "fixing" it with a sledgehammer that deindexed the pages it needed most. It teaches the chapter's limits — that the controls in §31.3 are precise instruments, and that the failure mode cuts both ways: index too much junk, or accidentally block your real catalog.

This is a labeled composite. It is not one named company; it is a pattern this book's narrator has seen play out repeatedly, assembled into one clean sequence. Every number below is illustrative — the SHAPE is the lesson, not the digits.

Background: a healthy store with a hidden overhang

Picture a constructed mid-size retailer — call it a home-and-kitchen store with about 9,000 products and a solid, growing stream of organic traffic. On the surface, everything is fine: good products, decent product pages, category pages that rank respectably. Under the surface, the store's platform has been quietly generating a parameter URL for every filter and sort combination a shopper could click — color, material, price band, brand, rating, plus four sort orders and three "items per page" views. Nobody configured this deliberately; it is simply what the platform does out of the box.

By the time anyone looks, Google has discovered roughly 1.4 million URLs for a 9,000-product store, and indexed about 220,000 of them — the overwhelming majority thin filter pages and duplicate sort-order variants of real category pages. The store's genuine assets — 9,000 products and a few hundred categories — are a rounding error inside a swamp of junk.

The first failure: the overhang starts to cost real money

For a while, the bloat is invisible, which is exactly why it is dangerous. Then two things happen.

First, crawl efficiency collapses. Googlebot spends most of its visits fetching ?sort= and ?color= permutations, so newly added products and updated categories get crawled slowly — new inventory takes a week or more to appear (a discovery/crawl delay, Chapter 1; crawl budget, Chapter 33). Second, a broad core update lands, and the store's rankings soften across the board — not a targeted penalty, but the quality dilution of a site whose indexed footprint is 95% thin pages (§31.4). Nobody can point to a single dropped page; the whole site just feels heavier and ranks a little worse everywhere.

text FIGURE C2.1 — "The overhang, in one snapshot" [constructed teaching example] Products in catalog ................................ ~9,000 Categories ........................................ ~400 URLs Google has DISCOVERED ........................ ~1,400,000 URLs INDEXED ...................................... ~220,000 Share of Googlebot requests hitting `?`-parameter URLs ... ~85% New product time-to-index ......................... 7–12 days (was <1 day) (All figures illustrative. The RATIO — 24× more indexed URLs than products — is the tell.)

The second failure: the fix that made it worse

Here is where the case earns its title. The team, now alarmed, reaches for the biggest hammer it can find and swings hard. In one deployment, it:

  1. Adds Disallow: /*?* to robots.txt — blocking Google from crawling any URL with a query string, to "stop wasting crawl budget."
  2. At the same time, adds noindex tags to all the filter pages — reasonably intending to remove them from the index.
  3. And, in the same "spring cleaning," 301-redirects 3,000 discontinued products straight to the homepage.

Every one of those moves is a mistake, and together they compound:

  • The robots.txt block cancels the noindex. Because robots.txt now forbids crawling any ?-parameter URL, Googlebot can no longer fetch those pages — so it never reads the noindex the team just added (§31.3). The 220,000 junk pages don't get cleanly de-indexed; many linger as bare, description-less listings, and Google is now blind to the instruction meant to remove them. The two controls were applied in the wrong order, and they neutralized each other.
  • Real pages get caught in the blast. Several important category and search-landing pages used a query-string parameter in their URLs (a faceted "ultralight" landing page lived at /tents?type=ultralight rather than a clean path). The blanket Disallow: /*?* blocked those too — so pages that were ranking and earning revenue abruptly stopped being crawled and began to fade. The sledgehammer hit the good pages along with the junk.
  • The homepage redirects become soft 404s. Google recognizes 3,000 redirects from unrelated product URLs to the homepage as irrelevant, treats them as soft 404s (§31.5), and passes essentially no equity — so the discontinued products' accumulated links and rankings evaporate instead of transferring anywhere useful.

Traffic, already soft, drops harder. The "fix" made three problems worse and solved none cleanly.

📄 Read the Report

text FIGURE C2.2 — "Why the fix backfired" [constructed teaching example] THE ACTION WHAT THE TEAM INTENDED WHAT GOOGLE ACTUALLY DID robots.txt: Disallow /*?* Stop crawling junk facets Also stopped reading the noindex → junk stayed indexed; blocked real parameter-based landing pages too noindex on filter pages Remove them from the index Never seen (page can't be crawled) 301 all dead products → homepage "Tidy" redirect Treated as soft 404s; equity lost WHAT IT SHOWS The §31.3 controls are precise instruments applied in a specific ORDER; used as a blunt simultaneous hammer, they cancel each other and catch the wrong pages. THE LESSON In technical SEO, the *sequence* and *targeting* are the whole game. "Block everything" is not a strategy; it's a second outage.

The recovery: precision, in order

The store eventually recovers, but only by undoing the hammer and doing the patient version:

  1. Un-block crawling of the parameter URLs (remove the blanket robots.txt Disallow) so Google can once again read the pages — because you cannot de-index what you won't let Google see.
  2. Let the noindex do its job. With crawling restored, Googlebot re-fetches the filter pages, reads the noindex, and drops them from the index over the following weeks. Index count falls from ~220,000 toward the real catalog size.
  3. Canonicalize sort/view variants to their base category so duplicate orderings consolidate instead of competing.
  4. Promote the handful of genuinely high-demand facets ("ultralight," "cast iron," "under \$50") to clean, static, indexable subcategory pages with unique intros — and give them real URLs, not query strings, so they can never be caught by a parameter block again.
  5. Re-map the discontinued redirects from the homepage to the most relevant category or successor product, so equity finally has somewhere sensible to flow (§31.5, Chapter 21).
  6. Only then, once the junk is de-indexed, add a targeted robots.txt rule for the specific worthless parameters — to save crawl going forward — being careful it can't touch the real landing pages.

Crawl efficiency returns; new products index in a day again; and with the index reflecting the real catalog, the site's quality signal recovers over the following core-update cycles.

The lesson

Two lessons, braided together, both about limits:

  • The controls in §31.3 are scalpels, not sledgehammers, and order matters. noindex and robots.txt solve different problems and interfere when combined carelessly: to remove pages from the index you must let Google crawl them long enough to see the noindex; blocking first blinds Google to your own instruction. There is a correct sequence, and skipping it creates a second outage on top of the first.
  • Over-correction is its own failure mode. The chapter says good faceted-nav handling "cannot make a page rank — and over-aggressive blocking can hide pages you actually wanted indexed." This case is that sentence made concrete. The blunt fix deindexed revenue pages and stranded equity. In e-commerce SEO, the damage runs in both directions: too much junk indexed, or too much of the good stuff blocked. Precision — the right control, on the right URLs, in the right order — is the entire craft.

Connect it to the chapter. Everything here is §31.3 and §31.5 under stress. The store's first mistake was letting facets explode; its worse mistake was believing a bigger hammer was a better fix. Both come from treating a precise technical system as if it were a volume knob.

Discussion questions

  1. Walk the exact causal chain by which Disallow: /*?* prevented the noindex from working. Why is "block it and noindex it" a contradiction, and what is the correct order?
  2. The blanket block also hit real, revenue-earning landing pages that happened to use a query-string parameter. What does this suggest about how you should design the URLs of high-demand facet pages before you ever need to block anything? (Tie to §31.3 "Move one.")
  3. Redirecting 3,000 dead products to the homepage produced soft 404s. Construct the correct redirect policy for those 3,000 URLs, distinguishing the cases in §31.5's decision tree.
  4. The store's rankings first softened during a core update even before the bad fix. Was the store "penalized"? Distinguish a penalty from quality-signal dilution, and say why the distinction changes the remedy. (Chapter 6 develops this.)
  5. A stakeholder asks, "If precision matters this much, why not just leave faceted navigation completely alone?" Give the honest cost of doing nothing — and the honest cost of over-doing it — so the answer lands as calibration, not avoidance.