Appendix F — HTTP Status Codes, robots.txt, and Crawl-Control Reference

A compact, practical reference for the technical signals that control whether and how search engines crawl and index your pages. Nothing here is code you must write from scratch — it is the vocabulary you need to read what your site is doing and to give a developer a precise instruction. Cross-references point to the chapters that explain the why.

1. HTTP status codes that matter for SEO

When a browser or a crawler requests a URL, your server answers with a three-digit status code. These are the ones that affect SEO; most others you can ignore.

Code	Name	What it means for SEO
200	OK	The page loaded normally. This is what you want for every page you want indexed.
301	Moved Permanently	A permanent redirect. Passes ranking signals to the target; the correct choice for migrations and consolidations (Chapter 21). Google eventually treats the target as canonical.
302	Found (temporary)	A temporary redirect. Use only when the move is genuinely temporary; a 302 left in place where a 301 belongs can delay consolidation.
304	Not Modified	"Nothing changed since you last crawled." An efficiency signal that can help crawl budget on large sites (Chapter 14).
307 / 308	Temporary / Permanent Redirect	Stricter HTTP/1.1 equivalents of 302/301 (method-preserving). 308 behaves like a 301 for SEO purposes.
404	Not Found	The page doesn't exist. Normal and healthy for genuinely gone pages; too many 404s on important URLs means broken links or a botched migration. A 404 keeps being crawled occasionally.
410	Gone	"This page is permanently gone, on purpose." A stronger signal than 404; Google tends to drop 410 URLs from the index slightly faster. Use when you intend removal.
429	Too Many Requests	Your server is rate-limiting the crawler. Persistent 429s can suppress crawling.
500	Internal Server Error	A server failure. Sustained 500s on important pages are an emergency — Google may drop pages it can't fetch.
503	Service Unavailable	"Temporarily down — try later." The correct code to serve during planned maintenance (Chapter 21); it tells Google to come back rather than to drop the page. Add a `Retry-After` header when you can.

The soft-404 trap. A "soft 404" is a page that returns 200 OK but shows a "not found" / empty message, or a thin page Google decides is effectively empty. Google flags these in Search Console. Fix them by either making the page substantive or returning a real 404/410 (Chapter 14).

2. `robots.txt` — controlling crawling

A plain-text file at the root of your domain (https://example.com/robots.txt) that tells crawlers which paths they may or may not fetch. Governed by the Robots Exclusion Protocol (standardized as RFC 9309).

Core syntax

User-agent: *                 # which crawler these rules apply to (* = all)
Disallow: /admin/             # don't crawl anything under /admin/
Disallow: /*?sort=            # wildcard: don't crawl URLs containing ?sort=
Allow: /admin/public-guide    # exception carved out of a broader Disallow
Sitemap: https://example.com/sitemap.xml   # where the sitemap lives (absolute URL)

User-agent: names the crawler (Googlebot, Bingbot, or * for all). You can have multiple blocks.
Disallow: blocks a path prefix. Disallow: / blocks the entire site (the classic accidental site-killer — Chapter 1's Case Study 1.2).
Allow: re-permits a path inside a disallowed area (Google honors the most specific match).
* matches any sequence of characters; $` anchors the end of a URL (`Disallow: /*.pdf$).
Sitemap: can appear anywhere and points to your XML sitemap(s).

The single most important caveat

robots.txt controls crawling, NOT indexing. Disallowing a URL stops Google from fetching it — but if that URL is linked from elsewhere, Google can still index the URL itself (often as a bare link with no description) because it was never allowed to crawl the page and see a noindex. To keep a page out of the index, do not block it in robots.txt; let it be crawled and put a noindex on it (Chapter 14).

3. `noindex` and the robots meta directives — controlling indexing

Applied per page, either as a <meta> tag in the HTML <head> or as an X-Robots-Tag HTTP header (useful for non-HTML files like PDFs).

<!-- In the page's <head>: -->
<meta name="robots" content="noindex, nofollow">

# As an HTTP response header (e.g., for a PDF):
X-Robots-Tag: noindex

Directive	Effect
`noindex`	Keep this page out of the search index. (The page must be crawlable for Google to see this.)
`nofollow`	Don't follow the links on this page / don't pass signals through them.
`noarchive`	Don't show a cached copy.
`nosnippet`	Don't show a text snippet or video preview for this page.
`max-snippet:[n]`	Limit the snippet to n characters.
`max-image-preview:[setting]`	Control image-preview size (`none` / `standard` / `large`).
`noimageindex`	Don't index images on this page.
`unavailable_after:[date]`	Drop the page from the index after a date.

Page-level link attributes (on individual <a> tags, not the whole page) — see Chapter 22: rel="nofollow" (no endorsement), rel="sponsored" (paid/ad link), rel="ugc" (user-generated content).

4. The canonical tag — consolidating duplicates

When the same or very similar content is reachable at multiple URLs, the canonical tag tells Google which one is the "master" to index and credit (Chapter 14).

<link rel="canonical" href="https://example.com/preferred-url">

A page can point its canonical at itself (self-referencing canonical — good practice).
The canonical is a strong hint, not a command; Google can override an illogical one.
Use it for tracking-parameter duplicates, print versions, and syndicated content (point to the original).
It does not stop crawling and is not a substitute for noindex when you truly want a page gone.

5. Which tool for which job — a decision guide

The commonest technical-SEO confusion is choosing among these four. Pick by intent:

You want to…	Use	Not
Keep a page out of the index	`noindex` (page crawlable)	`robots.txt` Disallow (Google can't see the noindex)
Save crawl budget on truly worthless URLs at scale	`robots.txt` Disallow	`noindex` (still gets crawled)
Consolidate duplicate/near-duplicate URLs	`rel="canonical"`	`noindex` (throws the page away instead of merging it)
Permanently move a URL	`301` redirect	canonical (weaker) or leaving the old URL live
Permanently remove a page	`410` (or `404`)	`noindex` left forever (page lingers, crawled)
Take the site down for maintenance	`503` + `Retry-After`	`404`/`503`-less downtime (risks pages being dropped)

Rule of thumb: robots.txt = "don't look." noindex = "look, but don't keep." canonical = "these are the same; keep that one." 301 = "it moved — send everything there." 410 = "it's gone."

6. Where to verify all of this (free)

Google Search Console → URL Inspection (Chapter 27): shows, for any URL, whether it's indexed, what status code Google saw, which canonical Google chose, and whether a noindex or block is in effect.
Search Console → Page indexing report: aggregates why pages aren't indexed (blocked by robots.txt, excluded by noindex, duplicate, crawled-not-indexed, soft 404, server error).
robots.txt report / tester in Search Console: confirms Google can fetch your robots.txt and how it parses a given rule.
A crawler (Screaming Frog's free tier, or similar; Chapter 30): fetches your whole site and lists every status code, redirect chain, canonical, and noindex in one table — the fastest way to audit.

Keep this appendix beside Chapter 14 (fundamentals) and Chapter 21 (migrations); together they cover the technical controls that decide whether your best content is ever seen at all.