Chapter 14 — Further Reading

Read primary sources first. For technical SEO more than anywhere else, Google's own documentation is authoritative, current, and free — and it is where the reference answer lives when a blog post and reality disagree. The items below are grouped by how much you can trust them, in the spirit of the chapter.

Tier 1 — Canonical (start here; free and authoritative)

• Google Search Central — "Introduction to robots.txt" and "How Google interprets the robots.txt specification." The definitive reference for what robots.txt does, the exact matching rules, and — most importantly — the repeated, explicit statement that it controls crawling, not indexing. This is the source for §14.1.
• Google Search Central — "Block Search indexing with noindex." Google's own page on the noindex meta tag and the X-Robots-Tag, including the crucial warning that a noindex page must not be blocked by robots.txt or Google can't see the tag. The source for §14.4 and the chapter's marquee myth.
• Google Search Central — "Build and submit a sitemap" and "Manage your sitemaps." What sitemaps are for, the format, the 50,000-URL / 50MB limits, sitemap index files, and the honest note that a sitemap does not guarantee indexing. The source for §14.2.
• Google Search Central — "Consolidate duplicate URLs" and "How to specify a canonical with rel=canonical and other methods." Google's guidance on duplicate content (including that there is generally no penalty), the canonical tag as a signal Google may override, and the difference between your declared canonical and the Google-selected one. The source for §14.3.
• Google Search Central — "HTTP status codes, network, and DNS errors" and the retry/maintenance guidance (returning 503 for downtime). How Google treats each status code, soft 404s, and the correct handling of planned maintenance. The source for §14.5.
• Google Search Central — "Large site owner's guide to managing crawl budget." Google's own account of crawl capacity and crawl demand, and the explicit guidance that most sites (generally under ~10,000 URLs) don't need to manage crawl budget at all. The source for §14.6.
• Google Search Central — "Page Indexing report" documentation. What each status ("Crawled – currently not indexed," "Discovered – currently not indexed," "Indexed, though blocked by robots.txt," "Excluded by 'noindex' tag," and the rest) actually means. The source for §14.7.
• Google Search Central Blog — the July 2019 posts on formalizing the Robots Exclusion Protocol and ending support for unsupported robots.txt rules (noindex, nofollow, crawl-delay). The primary source for Case Study 1, straight from the team that made the change.
• RFC 9309 — "Robots Exclusion Protocol" (IETF, 2022). The formal internet standard that finally codified the 25-year-old robots.txt handshake. Short, readable, and the last word on the syntax.

Tier 2 — Reputable secondary (useful, but verify against Tier 1)

• Long-standing technical-SEO references maintained by established SEO software vendors (for example, vendor "learn" hubs and technical-SEO guides). These synthesize crawlability, indexability, canonicalization, and faceted-navigation handling well — but they are written by companies that also sell crawlers and audit tools, so read them for the concepts and confirm any claim about Google's behavior against Google's own docs.
• Bing Webmaster documentation on crawling, robots.txt, and sitemaps. A second search engine's public account is a good way to separate universal principles (crawl vs. index; sitemaps as discovery) from Google-specific detail.
• Screaming Frog SEO Spider — user guide and how-tos. Even before you own a paid crawler, the free tier and its documentation teach you to see your site as a crawler does — status codes, canonicals, noindex, redirect chains, and orphan pages, all in one view. (The tool itself returns in Chapters 30 and 38.)

Tier 3 — Foundational and contextual (for the curious)

• The original Robots Exclusion Protocol proposal (Martijn Koster, 1994). The 25-year-old informal standard that RFC 9309 eventually formalized — historical, short, and a reminder of how much of the web runs on old handshakes.
• Constructed teaching examples in this chapter: Rivertown's crawl-and-index audit (Figure 14.3); the faceted-navigation store composite (Case Study 2); all figures with illustrative numbers.

Suggested order

1. Read Google's "Introduction to robots.txt" and "Block Search indexing with noindex" back to back — together they are the chapter's central lesson, in Google's own words, in about twenty minutes.
2. Skim the Page Indexing report documentation with your own Search Console open beside it, matching each status label to your real pages.
3. Read the canonical / duplicate URLs guidance, then the status codes page — these two cover most of the day-to-day technical decisions you'll make.
4. If you run a large site, read the crawl budget guide next; if you don't, skip it and save the time.
5. For history and the resolution of this chapter's myth, read the July 2019 announcements and skim RFC 9309.

The habit to build: whenever you read a technical-SEO claim anywhere, ask "which tier is this, and could I confirm it against Google's own documentation?" In this corner of SEO especially, the primary source is unusually complete — and unusually ignored.