Case Study 1 — The Day Google Killed the robots.txt noindex (2019), and Why the Standard Never Had One

Type: Real, public, Tier-1 documented events. Every factual claim here comes from Google's own public announcements (Search Central / the Google Webmasters blog, July 2019) and the published internet standard (RFC 9309, 2022). No statistic is invented. Where a detail is a reconstruction of the community's reaction, it is labeled as such.

Background: a 25-year-old handshake that was never a law

When you visit almost any website's /robots.txt, you are reading a file whose format was proposed in 1994 by Martijn Koster and adopted, informally, by essentially the entire web. For twenty-five years the Robots Exclusion Protocol (REP) was the definition of a de facto standard: universally used, universally understood, and — crucially — never actually a formal standard. There was no official document that said exactly which directives a crawler must honor. Different crawlers implemented slightly different rules. And into that ambiguity, over the years, crept a handful of unofficial directives that some sites used and some crawlers partially supported — among them a Noindex: line inside robots.txt.

That unofficial Noindex: directive is the villain of this chapter's central myth. Because Google's crawler had, for a period, partially honored it, a folklore grew up that you could keep a page out of Google by writing Noindex: in robots.txt. It seemed to work often enough to spread. It was never reliable, never documented as supported, and never part of any standard — a classic piece of SEO folklore that survived because it sometimes did the thing people wanted.

In the summer of 2019, Google did two things, days apart, that ended the ambiguity.

The issue: standardize the protocol, then remove what was never in it

First, on July 1, 2019, Google announced it was working to make the REP an official internet standard — and, to prove its commitment, it open-sourced its own robots.txt parser (the actual C++ library Googlebot uses to read the file) on GitHub, so the whole industry could see exactly how Google interprets the rules. This was a genuine act of transparency: the reference implementation for the most-used crawler on earth, published for anyone to inspect.

Second, on July 2, 2019, Google announced that as of September 1, 2019, Googlebot would stop honoring unsupported and unpublished rules in robots.txt — explicitly naming noindex, nofollow, and crawl-delay as directives it would no longer obey inside that file. The reasoning was exactly the distinction this chapter is built on: robots.txt is a crawling-control file, and trying to smuggle indexing-control directives into it produced unreliable, contradictory behavior. To keep the ecosystem healthy, Google drew the line cleanly: robots.txt governs crawling; use the right tools for indexing.

And Google did not leave people stranded. The same announcement listed the supported alternatives for controlling indexing — the tools this chapter teaches:

FIGURE C14.1 — "What to use instead of robots.txt noindex"        [after Google's July 2019 announcement]
  GOAL                                   THE SUPPORTED TOOL
  Keep a page out of the index           noindex in a meta robots tag OR the X-Robots-Tag HTTP header
  Remove content that should be gone      Return 404 (Not Found) or 410 (Gone)
  Protect private content                 Password protection (authentication) — not robots.txt
  Stop crawling of a section              Disallow in robots.txt (crawl control — NOT index control)
  Remove something urgently               The Search Console Removals tool (temporary)

Read that table against the myth. Every legitimate goal people thought the robots.txt noindex served has a proper, supported tool — and the proper tool for keeping a page out of the index (a noindex on the page) requires that Google be allowed to crawl the page. The robots.txt noindex was folklore filling a gap that the real tools already covered.

📄 Read the Report

text FIGURE C14.2 — "Reading Google's 2019 announcements" [after Google Search Central, July 2019] THE SOURCE Google Webmasters/Search Central posts (July 1–2, 2019) + the open-sourced robots.txt parser on GitHub. WHAT'S THERE (1) Intent to standardize the REP; (2) the parser published for inspection; (3) an end date (Sep 1, 2019) for honoring noindex/nofollow/crawl-delay in robots.txt; (4) a list of supported alternatives. WHAT IT SHOWS Google formalizing the crawl-vs-index distinction in public: robots.txt = crawling; indexing has its own, separate tools. The most authoritative possible confirmation of this chapter's marquee rule. WHAT IT DOESN'T It does NOT mean sites using the old directive were penalized — they simply lost an unreliable behavior; and it does NOT change that a robots.txt-blocked URL can still be indexed from links. THE MOVE Audit any site you own for a leftover robots.txt `noindex` — since Sep 2019 it does nothing, so a page you *thought* was hidden may be fully indexable. Verify with the Page indexing report. THE LESSON When Google publishes the reference implementation and an end date, believe it over any blog post. The primary source ended a decade of folklore in two sentences.

What it shows

Three lessons sit inside this episode, and each one is a pillar of Chapter 14.

  1. The crawl-versus-index distinction is not a teaching simplification — it is the actual architecture, formalized by the people who built the crawler. Google didn't merely prefer that you not use robots.txt for indexing; it removed the behavior entirely and pointed you to separate tools, because the two jobs live at different stages of the pipeline (Chapter 1).

  2. Folklore can survive for years on partial, accidental confirmation. The robots.txt noindex "worked" often enough to spread and never reliably enough to trust. This is the exact pattern the book's evidence-tier habit (Chapter 2) exists to catch: a tactic that seems to work is not the same as a tactic that is documented to work. When Google's own parser and announcement contradict the folklore, the folklore loses.

  3. The most authoritative source is public, free, and sometimes literally open-source. Google published the actual code Googlebot uses to read robots.txt. In how many fields can you read the reference implementation of the single most important gatekeeper? A working SEO reads primary sources; here, the primary source is a GitHub repository and a dated blog post.

Outcome

The change took effect on September 1, 2019, with little drama — precisely because most well-run sites already used meta-robots noindex, not the robots.txt version. The sites that were relying on the unofficial directive faced a quiet risk: pages they believed were hidden could now be crawled and indexed normally, so the correct response was to audit for leftover robots.txt noindex lines and replace them with real page-level noindex tags. Three years later, in September 2022, the Robots Exclusion Protocol was published as RFC 9309, finally making the 25-year-old handshake an official internet standard — and codifying, once and for all, what robots.txt does (crawl control) and does not do (index control).

There is no villain and no traffic catastrophe in this story. That is the point. It is a case about epistemics — where reliable knowledge in SEO comes from — and it resolves in favor of the primary source over the folklore, every time.

The lesson

Use the right tool for the stage, and get your facts from the source that owns the mechanism. The robots.txt noindex was never a supported, standardized control; when Google finally said so in writing and published the code to prove it, a decade of confident blog posts became wrong overnight. The durable takeaways are the chapter's spine: robots.txt controls crawling; noindex (that Google is allowed to crawl and read) controls indexing; and when a tactic and Google's own documentation disagree, believe the documentation.

Discussion questions

  1. For years, the robots.txt noindex "worked" for some sites and not others. Using the book's evidence-tier framework (Chapter 2), how should a practitioner have classified this tactic before Google's 2019 announcement — and what would have kept them from relying on it?
  2. Google open-sourced its robots.txt parser. Why is publishing the reference implementation a stronger form of documentation than a prose description of the rules? What ambiguity does it remove?
  3. The 2019 change created a hidden risk: pages that owners believed were de-indexed via robots.txt were suddenly indexable. Draft the exact audit step you would run to find such pages on a site you inherited.
  4. Google removed crawl-delay support at the same time as noindex. Given what you learned about crawl budget (§14.6), why might Google prefer to manage crawl rate itself rather than honor a site-declared delay?
  5. The Robots Exclusion Protocol was a de facto standard for 25 years before becoming RFC 9309 in 2022. What are the risks, for an entire industry, of building critical infrastructure on an informal standard that different vendors interpret differently — and how does this connect to the chapter's warning that each crawler may treat robots.txt slightly differently?