Chapter 14: Technical SEO Fundamentals — Crawlability, Indexability, and Making Sure Google Can See Your Site

"You cannot rank a page a search engine cannot see. Half of technical SEO is making sure that sentence never applies to a page you want found — and the other half is making sure it does apply to the pages you don't." — a working principle for this chapter [constructed]

Overview

Here is the uncomfortable question that opens Part III, and it comes before any question about content, keywords, or links: can Google actually see this page — find it, fetch it, and file it — at all?

Most site owners never ask it. They assume that because a page loads in their browser, it exists for Google too. That assumption is wrong often enough to be dangerous, and when it is wrong it is invisible: the page looks perfect to a human, ranks for nothing, and no amount of rewriting, keyword tuning, or link building will help — because none of those things touch the actual break. The page was never in the race. In Chapter 1 we walked the five-stage pipeline — discover, crawl, render, index, rank — and made the central promise of this book: every SEO problem is a break somewhere in that chain. This chapter is where you learn to inspect and repair the earliest, most consequential links in it: crawlability (can Google fetch your pages?) and indexability (will Google keep them?).

The tools are unglamorous and they are the plumbing of the whole discipline. A robots.txt file that tells crawlers where they may and may not go — and quietly backfires when people ask it to do a job it cannot do. An XML sitemap that helps Google discover your URLs but guarantees nothing. Canonical tags that tell Google which of several near-identical URLs is the real one. Meta robots tags that say "index this" or "keep this out." The HTTP status codes your server returns, each of which is a message to the crawler. Crawl budget — the topic everyone worries about and almost nobody needs to. And Search Console's indexing reports, the honest mirror that tells you, page by page, whether Google can see your site.

None of this is a trick. This is the theme that runs under the entire chapter: these controls exist so you can help Google understand your site accurately — which of your URLs matter, which are duplicates, which are gone, which to ignore. Used well, they are cooperation. Used carelessly, they are how a good site accidentally hides from the one audience it was built for.

In this chapter, you will learn to:

• Read and reason about a robots.txt file, and recognize the ways it turns into a self-inflicted wound.
• Tell apart the four controls people constantly confuse — block crawling, block indexing, consolidate duplicates, and move a page — and pick the right one.
• Treat a sitemap as a map, not a magic wand.
• Interpret the status codes a server returns and what each means for how Google treats the URL.
• Decide, with evidence, whether crawl budget is your problem (it probably isn't).
• Open Search Console's Page indexing report and turn each status label into a specific next action.
• Hunt down the most catastrophic and most common technical error in SEO: an accidental noindex.

Learning Paths

Part III's plumbing chapter is for everyone, but weight it to your work. 🔧 Developer: this is your bread and butter — read every section closely; §14.4 (meta robots vs. X-Robots-Tag) and §14.5 (status codes) are yours to own and to explain to non-technical colleagues. 📊 Strategist: treat the whole chapter as the first pass of any technical audit (Chapter 38); the diagnostic mindset in §14.7 is what you will run on every site you ever touch. 🛒 E-Commerce: §14.3 (canonicals and duplicate content) and §14.6–14.7 (crawl budget and index bloat) matter more to you than to almost anyone — faceted navigation makes duplicate and thin URLs at scale, and that is Chapter 31's deep dive. 🏪 Local Business: focus on §14.1 and §14.7 — the accidental-noindex hunt in the Strategy File is the single highest-value thing in this chapter for a small site on WordPress. 📝 Content Creator: §14.2 (sitemaps) and §14.7 (why a good post shows "Crawled – currently not indexed") will explain more of your indexing frustration than any content advice.

14.1 robots.txt: directing the crawler (and the ways it backfires)

Every well-behaved crawler that visits your site does one thing before anything else: it asks for a single file at the root of your domain, at https://yourdomain.example/robots.txt. That file is your robots.txt — a plain-text file, following the long-standing Robots Exclusion Protocol (formalized in 2022 as the internet standard RFC 9309), that tells crawlers which parts of your site they are and are not welcome to fetch. It is the first handshake between your server and Googlebot, and understanding exactly what it can and cannot do prevents a startling share of SEO disasters.

The syntax is simple. A robots.txt file is a list of rules grouped by which crawler they apply to:

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /search
Allow: /wp-admin/admin-ajax.php

Sitemap: https://rivertownhome.example/sitemap_index.xml

Here is what that says, line by line. User-agent: * means "the following rules apply to all crawlers." Each Disallow: line names a path Googlebot should not fetch — here, the WordPress admin area, the shopping cart, and internal search-results URLs. The Allow: line carves out one exception inside a disallowed folder. And the Sitemap: line points crawlers to your sitemap (more on that in §14.2). A crawler reads this, and — if it is a compliant crawler like Googlebot — it obeys.

Now the single most important sentence in this chapter, the one that prevents the most damage: robots.txt controls crawling, not indexing. It tells Google what it may fetch. It does not, and cannot, reliably tell Google what to keep out of search results. Those are two different stages of the pipeline (crawl versus index, from Chapter 1), and conflating them is the source of nearly every robots.txt catastrophe.

🔎 How Search Sees It When Googlebot wants to crawl a URL, it first checks its cached copy of your robots.txt (Google re-fetches it periodically, roughly daily). If a rule disallows that URL, Googlebot will not fetch the page's contents. But here is the twist that surprises people: Google can still index a URL it was told not to crawl. If other pages link to that blocked URL, Google knows the URL exists — it just can't look inside it. So it may list the bare URL in search results with no description, showing something like "No information is available for this page." You blocked the crawl and got the opposite of what you wanted: the page is in the index, but as an empty, uninformative husk you no longer control. Search Console has a specific status for exactly this — "Indexed, though blocked by robots.txt" — and it is one of the most common confused-signals problems in all of technical SEO.

That mechanism is the root of the marquee myth of this entire chapter — the one every practitioner must internalize.

🚫 SEO Myth: "To keep a page out of Google, add a noindex to your robots.txt." You cannot. This is the most consequential misunderstanding in technical SEO, and it fails for two stacked reasons. First, robots.txt has no reliable noindex directive. Some sites once used an unofficial Noindex: line, but it was never part of the standard, and Google confirmed in 2019 that it does not support it (it stopped honoring the unofficial rule as of September 2019). Second, and more fundamentally: the correct way to keep a page out of the index is a noindex instruction on the page itself (§14.4) — and for Google to see that instruction, it has to crawl the page. If you Disallow the page in robots.txt, you have blocked the very crawl Google needs in order to read your noindex. The two controls fight each other. The page stays crawl-blocked, Google never sees your "keep me out" instruction, and — if anything links to the URL — it can sit in the index indefinitely as a bare link you cannot remove. The rule to carve into memory: to de-index a page, you must allow it to be crawled so Google can read the noindex you put on it. Block crawling and you blindfold the very inspector whose verdict you were trying to change.

So what is robots.txt genuinely good for? It is good for crawl management — steering Googlebot away from parts of your site that have no business being crawled and would only waste effort: infinite calendar URLs, internal search-results pages, faceted-navigation parameter explosions (Chapter 31), admin areas, staging paths. On a large site, keeping the crawler out of junk lets it spend its attention on pages that matter (§14.6). What it is not good for: keeping a page out of search results (use noindex), hiding sensitive or private content (the file is publicly readable — anyone can visit yourdomain.example/robots.txt and see exactly which folders you tried to hide; use real authentication instead), and it is not a security boundary of any kind.

🛠️ Try It on Your Site Type your domain followed by /robots.txt into a browser — for example rivertownhome.example/robots.txt. Every public site has one (or returns a 404, which crawlers treat as "crawl everything"). Read it line by line. Is there a Disallow: / on its own? That single line blocks your entire site from crawling — a five-alarm fire if it is on a live production site (it is the classic leftover from a staging environment). Do you see anything important disallowed by accident — your blog, your product folder, your /wp-content/ (which can block the CSS and images Google needs to render the page)? Do you see a Sitemap: line? You have just performed the first check of a technical audit, and it took thirty seconds.

The failure modes of robots.txt are worth stating as a checklist, because each has sunk real sites: a stray Disallow: / blocking everything; disallowing resource folders (CSS/JavaScript) so Google can't render the page properly; using it to try to de-index (see the myth above); assuming it hides content (it doesn't); and forgetting that each subdomain and protocol needs its own file (blog.yourdomain.example/robots.txt is separate from yourdomain.example/robots.txt). robots.txt can shape where the crawler spends its time; it cannot remove a page from search, protect anything, or override links pointing at a blocked URL.

14.2 XML sitemaps: helping Google discover your pages

If robots.txt is the sign that says where crawlers may not go, an XML sitemap is the directory that says "here are the pages I want you to know about." It is a structured file — in the XML format machines read easily — that lists the URLs on your site, optionally annotated with the date each was last modified. You submit it to Google (through Search Console) and reference it in your robots.txt, and Google uses it as one input into discovery: the first stage of the pipeline, finding out that your URLs exist.

A sitemap entry looks like this:

<url>
  <loc>https://rivertownhome.example/services/water-heater-replacement/</loc>
  <lastmod>2026-06-14</lastmod>
</url>

That is the whole idea: a URL and, ideally, an honest "last modified" date. You will also see <changefreq> and <priority> tags in older sitemaps, but Google has stated it ignores both — they were well-intentioned and became noise, so don't spend a second on them. The <lastmod> date Google will use, but only if you keep it accurate; a sitemap where every page claims to have changed today, every day, teaches Google to distrust the field entirely.

The critical thing to understand about a sitemap is what it does not do, because this is where hope outruns reality.

🚫 SEO Myth: "If I put a page in my sitemap, Google will index it." No. A sitemap is a discovery aid, not an indexing command. Listing a URL tells Google the page exists and invites a crawl; it does not force Google to crawl the page, and it certainly does not force Google to keep it. Google decides indexing on the merits (the quality gate from Chapter 1), and it routinely ignores sitemap URLs it judges thin, duplicate, or low-value — you will see them later as "Crawled – currently not indexed" or "Discovered – currently not indexed" (§14.7). A sitemap is also not a ranking factor: being in it earns you no ranking advantage. It helps Google find your pages faster and more completely — genuinely valuable for new sites, large sites, sites with weak internal linking, and pages with few inbound links — but "found" is a long way from "ranked."

Who actually benefits from a sitemap, and how much? The honest answer scales with your situation. A small, well-linked site where every page is two clicks from the homepage barely needs one — Google will find those pages by following links regardless. A large site, a brand-new site with no inbound links yet, a site with deep or orphaned pages, or a site with content types Google discovers poorly (a video library, a news archive) benefits substantially. In all cases a sitemap is cheap insurance, and most modern platforms generate one automatically — so there is little reason not to have a clean one.

That word clean is doing real work. A good sitemap is a curated list of your canonical, indexable, live URLs — pages that return a 200 OK status (§14.5), are not blocked by robots.txt, do not carry noindex, and are the canonical version (§14.3). A sitemap stuffed with redirects, 404s, noindex pages, and non-canonical duplicates sends Google a confused, self-contradicting message and erodes its trust in the file. If your sitemap says "index this" while the page itself says "don't," you have created exactly the kind of mixed signal this chapter teaches you to eliminate.

🛠️ Try It on Your Site Find your sitemap. Try yourdomain.example/sitemap.xml, yourdomain.example/sitemap_index.xml, and check the Sitemap: line in your robots.txt. On WordPress you likely have one automatically — the core software has generated wp-sitemap.xml since version 5.5, and SEO plugins like Yoast or Rank Math generate their own sitemap_index.xml. Open it and spot-check: are these the URLs you actually want indexed? Do you see junk (tag archives, author pages, attachment URLs, old campaign pages)? A messy sitemap is often the first visible symptom of the index-bloat problem we reach in §14.7. (One caution for WordPress users: make sure you are not submitting two competing sitemaps — the core one and the plugin one — pick the plugin's and let it manage the file.)

A few mechanical limits, mostly for larger sites: a single sitemap file can hold up to 50,000 URLs and be no larger than 50MB uncompressed; beyond that you split into multiple sitemaps and list them in a sitemap index file (a sitemap of sitemaps). There are specialized sitemap types for images, video, and news. And you submit the sitemap in Search Console's Sitemaps report, where Google will tell you when it last read the file and how many URLs it found — the beginning of the feedback loop we close in §14.7. A sitemap can accelerate and complete discovery, especially at scale; it cannot get a page crawled on demand, force indexing, or improve rankings.

14.3 Canonical tags and duplicate content: one page, one URL

Here is a fact about the web that causes more quiet SEO trouble than almost anything else: the same page can live at many different URLs. Consider Rivertown's water-heater page. Every one of these might serve identical content:

ONE PAGE, MANY URLS                                     [schematic — not to scale]
  https://rivertownhome.example/services/water-heater-replacement/
  http://rivertownhome.example/services/water-heater-replacement/     ← http, not https
  https://www.rivertownhome.example/services/water-heater-replacement/ ← www
  https://rivertownhome.example/services/water-heater-replacement      ← no trailing slash
  https://rivertownhome.example/services/Water-Heater-Replacement/     ← different case
  https://rivertownhome.example/services/water-heater-replacement/?utm_source=facebook
  https://rivertownhome.example/services/water-heater-replacement/?ref=newsletter

To a human these are "the same page." To Google, each distinct URL is, by default, a distinct page — and now you have seven copies of one thing competing with each other. This is duplicate content: substantially identical or near-identical content that is accessible at more than one URL. It arises constantly and mostly innocently: from http/https and www/non-www variants, trailing-slash and capitalization differences, tracking parameters on shared links, printer-friendly versions, session IDs, product variants (a shirt in six colors on six URLs), tag and category archives on a blog, and — the giant of e-commerce — faceted navigation, where every filter combination mints a new URL (Chapter 31).

Before we fix it, let's kill the fear that surrounds it.

🚫 SEO Myth: "Duplicate content gets you a Google penalty." In almost every ordinary case, no. Google has stated plainly and repeatedly that there is no "duplicate content penalty" for the routine, non-malicious duplication described above. What actually happens is quieter and non-punitive: when Google finds duplicates, it picks one version to index and show, and folds the others into it. The real costs are three, and none is a penalty: (1) Google might choose a different "winner" than you would have — indexing the ?utm_source=facebook version instead of your clean URL; (2) ranking signals get split across the copies instead of concentrating on one strong page (links pointing at three URL variants are three weaker votes instead of one strong one); and (3) on a large site, crawl effort is wasted re-fetching copies. A penalty is reserved for deliberately deceptive duplication — scraped content, spun articles, doorway pages (Chapter 33's caution). Your accidental URL variants are not that. So: don't panic about duplicate content — manage it, so Google consolidates on the URL you chose.

The primary tool for that management is the canonical tag: a line in a page's HTML <head> that names the preferred ("canonical") URL for that content.

<link rel="canonical" href="https://rivertownhome.example/services/water-heater-replacement/" />

Placed on all seven URL variants above, this tag tells Google: "whichever of these you crawled, treat this clean URL as the real one; consolidate all the signals here." Every variant points at the same canonical, and Google concentrates ranking signals and indexing on that one. Best practice is that your canonical page also points at itself (a "self-referencing canonical"), removing all ambiguity. For non-HTML files like PDFs, where there is no <head>, the same instruction can be sent as an HTTP header instead.

But — and this is the limit every practitioner must respect — the canonical tag is a hint, not a command.

🔎 How Search Sees It You declare a canonical; Google decides one. Google treats your canonical tag as a strong signal, but it weighs it against everything else it knows: your internal links, your sitemap entries, any redirects, hreflang annotations (Chapter 20), and the actual similarity of the pages. If those signals contradict your tag — if you canonicalize page A to page B but every internal link and your sitemap point to A, or if A and B aren't actually that similar — Google may override you and pick its own "Google-selected canonical," which can differ from your "user-declared canonical." You can see both, side by side, in Search Console's URL Inspection tool. When they disagree, Google is telling you your signals are muddled. The fix is never to fight the tag harder; it is to make all your signals agree — canonical, links, sitemap, and redirects all pointing the same way.

📄 Read the Report

text FIGURE 14.1 — "Google chose a different canonical" [constructed teaching example] THE QUERY / PAGE URL Inspection in Search Console for /services/water-heater-replacement/?utm_source=facebook WHAT'S THERE Status: "Alternate page with proper canonical tag." User-declared canonical: the clean /services/water-heater-replacement/ URL. Google-selected canonical: the same clean URL. The tracking-parameter version is correctly folded in. WHAT IT SHOWS The canonical setup is working: signals point at the clean URL, and Google is consolidating the tracking variants onto it rather than indexing seven copies. WHAT IT DOESN'T It does NOT prove every parameter variant is handled — only the one inspected; and it says nothing about whether the clean page ranks well, only that it's the chosen one. Consolidation is hygiene, not a ranking win. THE MOVE Spot-check a few more parameter and case variants. If any shows "Duplicate, Google chose different canonical than user," align its links and sitemap entry. THE LESSON Your job is to make one URL the obvious winner. When your signals agree, Google agrees. When they conflict, Google decides — and you may not like the choice.

Canonical is not the only consolidation tool, and choosing the wrong one is a common error. Keep them straight: a canonical tag says "these are duplicates; index this one, but both URLs may stay reachable" — right for genuine duplicates you want to keep accessible (tracking-parameter versions, print pages, product variants). A 301 redirect (§14.5) says "this URL is gone, permanently send everyone to the new one" — right when a URL should truly cease to exist. A noindex (§14.4) says "keep this specific page out of the index entirely" — right for pages that shouldn't rank at all (a thank-you page, an internal filter result). And robots.txt says "don't even crawl here." They are four different jobs, and the canonical tag can consolidate duplicates and pool their signals; it cannot force Google's hand, secure private content, or substitute for a redirect when a page has actually moved.

14.4 Meta robots and X-Robots-Tag: noindex and nofollow

We have now referenced noindex several times as the correct way to keep a page out of search. It is time to define it precisely. The meta robots tag is a line in a page's HTML <head> that gives page-level instructions directly to search engines:

<meta name="robots" content="noindex, follow" />

The two values you will use most are the reason this tag exists. noindex tells Google: do not include this page in the index — do not show it in search results, ever, even if it's crawled and even if it's linked. nofollow, at the page level, tells Google not to follow the links on this page or pass any signal through them. You can combine them. The most common and useful combination is noindex, follow: "keep this page out of search, but do still crawl through its links to discover and credit the pages it points to." (A subtle note Google has made over the years: on a page it has decided to drop from the index and rarely re-crawls, it may eventually treat the links as effectively unfollowed anyway — so noindex, follow is best understood as your stated intent, not a permanent guarantee about link-following on a page Google has stopped visiting.)

What is noindex for? For the many pages a real site has that are useful to humans but have no business ranking: internal search-results pages, thank-you and confirmation pages, thin tag or filter archives, staging or duplicate pages you want reachable but invisible, login pages, "print this" versions. Applying noindex to these keeps your index clean and focused — the direct antidote to the index-bloat problem in §14.7.

Now, the crucial mechanical fact, which connects straight back to §14.1's myth:

🔄 Check Your Understanding You want to remove an old promotional landing page from Google. A colleague suggests adding Disallow: /old-promo/ to robots.txt. Why is that the wrong tool, and what should you do instead?

Answer

robots.txt blocks crawling, and to obey a noindex Google must crawl the page to read it. If you Disallow the page, Google can never see the noindex, so the page can stay in the index (as a bare, uninformative link if anything points to it). The right move: allow crawling and put a <meta name="robots" content="noindex"> on the page itself. Let Google crawl it, read the "keep me out" instruction, and drop it. Only after it has been de-indexed could you consider blocking the crawl to save crawl effort — but usually you just leave the noindex in place. Block-then-noindex is the classic self-defeating combination.

There is a second delivery mechanism for the same instructions, and developers need it: the X-Robots-Tag. This sends noindex, nofollow, and their relatives as an HTTP response header rather than an HTML tag. Why would you need that? Because some things you want to control aren't HTML pages and have no <head> to put a meta tag in — PDFs, images, spreadsheets, generated files. The X-Robots-Tag lets your server attach a noindex to a PDF, or to an entire folder of files, via server configuration. It is functionally identical to meta robots for the crawler; it is just the way to reach non-HTML content. (Everything about crawlability still applies: for Google to read an X-Robots-Tag header, it has to be allowed to fetch the file.)

Beyond noindex and nofollow, the robots meta tag supports other useful directives worth recognizing: noarchive (don't show a cached copy), nosnippet (don't show a text snippet), max-snippet, max-image-preview, and max-video-preview (limit how much preview Google shows), and unavailable_after (drop from the index after a set date — handy for time-limited event or job pages). You will rarely need most of them, but knowing they exist means you can reach for the right one.

One boundary to mark clearly, because two different nofollows exist and this chapter owns only part of the story. The page-level meta nofollow above is not the same as the link-level rel="nofollow" attribute you put on an individual <a> link. Google has, since 2019–2020, treated link-level rel="nofollow" (and its cousins sponsored and ugc) as hints rather than strict directives, and the whole question of which links to nofollow and why is a link-strategy topic, not a crawlability one.

🔗 Connection The link-level rel="nofollow", rel="sponsored", and rel="ugc" attributes — what they mean for how authority flows and when to use them — are covered in Chapter 22 (How Links Work). Here in Chapter 14 we care only about the page-level robots directives that control crawling and indexing. Two different tools, similar names; don't confuse them. And for the tool that shows you exactly how Google is treating a given URL — the URL Inspection tool — see Chapter 27 (Google Search Console).

Meta robots and X-Robots-Tag can keep a page out of the index cleanly and control snippet/preview behavior; they cannot work if Google is blocked from crawling the page (§14.1), and noindex is not a way to hide content from determined humans — it only governs search engines.

14.5 The HTTP status codes that matter

Every time Googlebot (or a browser) requests a URL, your server answers with a three-digit HTTP status code before it sends anything else. That code is a message about what happened, and Google reads it as an instruction about how to treat the URL. You do not need all of them — there are dozens — but a working SEO must fluently read the handful that shape crawling and indexing. Here they are, with what each one tells Google to do.

Read down that table and you can see the logic of each pair. The two redirects — 301 and 302 — differ only in permanence, but that difference is consequential. A 301 (permanent) consolidates the old URL's accumulated ranking signals onto the new one and tells Google to replace the old URL with the new; a 302 (temporary) deliberately keeps the old URL in the index because you're saying "I'll be back." Using a 302 when you mean a 301 — a shockingly common misconfiguration, since many systems default to 302 — can slow or prevent the consolidation you wanted. (Google has said that a long-lived 302 will eventually be treated like a 301, but "eventually" is not a strategy; return the code you mean.) The redirect discipline — mapping every old URL to its best new equivalent, avoiding chains — is the heart of Chapter 21 on migrations; here, just know which status code carries which meaning.

The two "this is gone" codes — 404 and 410 — differ only in how emphatic they are. A 404 (Not Found) is the everyday "no page here"; a 410 (Gone) adds "and I mean permanently." Both are completely normal, which brings us to another persistent myth.

🚫 SEO Myth: "404 errors hurt your site's rankings, so you must fix every one." Mostly false, and the panic wastes real time. Google has said repeatedly that returning a 404 for a page that genuinely no longer exists is normal and expected — the web is full of dead links, and crawling 404s does not harm the ranking of the rest of your site. A page that's gone should return 404 (or 410). The things that do matter around 404s are different and specific: (1) a 404 on an important URL that people and other sites still link to is a lost-traffic and lost-signal problem — that URL should be 301-redirected to a relevant live page, not left to 404; and (2) a soft 404 is a genuine defect. A soft 404 is a page that shows "not found" or is effectively empty but returns a 200 OK status — lying to Google that a real page exists where there is none. Google detects and flags these, and they waste crawl and clutter your index. So: let genuinely dead pages 404; redirect valuable dead URLs with a 301; and hunt down soft 404s (a "no results" page returning 200, a deleted product showing a bare template) and make them return the honest code.

The two server-trouble codes — 500 and 503 — are where getting the code right protects your site during its most vulnerable moments. A 500 (Internal Server Error) says something broke; if Googlebot hits many 500s, it slows its crawl (to avoid making things worse) and, if the errors persist for a long time, can begin dropping the affected pages from the index. A 503 (Service Unavailable) is the responsible code for temporary downtime — planned maintenance, a deploy, an overloaded moment. It tells Google "not now, come back later," ideally paired with a Retry-After header, and Google will pause rather than treat the outage as a permanent loss.

⚖️ Evidence Check Claim: "Serving 503 during maintenance protects your rankings; serving 200 or 500 can hurt them." — Confirmed by Google: Google's own documentation explicitly recommends returning 503 for temporary unavailability and warns against serving a maintenance page with a 200 status (which tells Google the broken/placeholder page is your real content and can get it indexed). This is Tier-1, straight from Search Central. — The honest caveat: the magnitude of harm from a brief 200-during-maintenance slip is not something anyone can quote a precise number for — a few minutes rarely matters; hours or days of the wrong code during a busy crawl window matters more. What's certain is the direction and the correct action: use 503 for planned downtime. The exact damage from doing it wrong is situational, and anyone who gives you a precise percentage is guessing.

Status codes can tell Google precisely how to treat a URL — keep it, move it, forget it, or wait; they cannot fix a quality or relevance problem, and the wrong code (a 302 meant as 301, a soft 404, a 200 on a broken page) actively misleads Google in ways that are entirely self-inflicted.

14.6 Crawl budget: who actually needs to care (most don't)

Crawl budget — a term we met in Chapter 1 and can now examine properly — is, loosely, the number of pages Googlebot will crawl on your site in a given period. It is worth understanding, and it is worth not worrying about, and holding both of those at once is the mark of someone who actually understands it. Let's do that.

Google's own framing has two components. First, a crawl capacity limit: how much crawling your server can tolerate without slowing down. Google monitors your server's responses, and if pages start getting slow or returning errors, it backs off to avoid harming your site; if your server is fast and healthy, it will crawl more freely. Second, crawl demand: how much Google wants to crawl you, which rises with your site's popularity and importance and with how often your content genuinely changes, and falls for stale, obscure, or low-value URLs. Your effective crawl budget is roughly where those two meet — capacity you can support, times demand Google has.

Now the honest, deflating truth, which we planted in Chapter 1 and now confirm with Google's own guidance:

🚫 SEO Myth: "I need to optimize my crawl budget." (revisited, with the evidence) For the overwhelming majority of sites, crawl budget is a solution in search of a problem. Google's own large-site guidance states, in effect, that sites with up to a few thousand URLs — and generally anything under roughly 10,000 URLs — do not need to worry about crawl budget management; Google will crawl them efficiently as a matter of course. The topic is over-discussed precisely because it sounds important and technical. If your site is a few hundred or a few thousand pages on a reasonably fast server, Googlebot is not struggling to keep up with you, and hours spent "optimizing crawl budget" are hours stolen from content, links, and the fixes that actually move the needle. Crawl budget is a real constraint for a real, identifiable minority: very large sites (hundreds of thousands to millions of URLs), sites that auto-generate huge numbers of URLs (faceted navigation, parameters, calendars), and sites with frequent large-scale changes. Those owners usually know who they are.

If you are in that minority, the levers are the ones this whole chapter has been building: reduce the crawling of junk so Google spends its attention on what matters. Block genuinely worthless URL spaces in robots.txt (§14.1); eliminate duplicate URLs with canonicals and clean URL structures (§14.3); remove soft 404s and long redirect chains (§14.5); consolidate or noindex thin pages (§14.4, §14.7); and keep your server fast so Google's capacity limit stays high. None of that is exotic — it is technical hygiene, applied at scale.

🛠️ Try It on Your Site Even if you're sure crawl budget isn't your problem, confirm it with evidence rather than faith. In Search Console, open Settings → Crawl stats. You'll see total crawl requests over time, average server response time, and host status. Two quick reads: is your average response time low (fast server, healthy capacity)? And is the number of pages Google crawls comfortably larger than your number of real pages (Google is keeping up)? If a 400-page site shows Google making tens of thousands of crawl requests, something is generating URLs you don't know about — a parameter or duplicate problem worth finding. For most readers, this report will simply reassure you that crawl budget is not your bottleneck, and you can stop thinking about it.

🔗 Connection The deep treatment of crawl budget — log-file analysis (reading your server logs to see exactly what Googlebot fetched), crawl-budget optimization for genuinely large sites, and index management at scale — lives in Chapter 33 (Programmatic and Enterprise SEO). If you run a site big enough for crawl budget to matter, that's your chapter. If you don't, the best crawl-budget strategy is to stop thinking about it and go improve a page.

One honesty note to close: crawl budget is not a ranking factor. Crawling more of your site does not raise its rankings; it only affects how quickly and completely your pages get discovered and refreshed. So even where crawl budget genuinely matters, its payoff is faster indexing of new and updated content — not a ranking boost. Crawl-budget management can help a very large site get its important, fresh content crawled promptly; it cannot improve rankings directly, and for small and mid-size sites it is a non-issue you can confirm and forget.

14.7 Search Console's indexing reports: the honest mirror

Everything in this chapter converges here, in the free tool that tells you the truth: Google Search Console (GSC) and its Page indexing report. This is where you stop guessing about whether Google can see your site and start reading Google's own verdict, page by page. Chapter 27 is the full tour of Search Console; here we focus on the indexing reports specifically, because they are the payoff of crawlability and indexability.

The Page indexing report splits your known URLs into two buckets — Indexed and Not indexed — and, for the not-indexed pages, gives a reason for each. Learning to read those reasons is the single most practical skill in technical SEO, because each reason maps to a specific stage and a specific action. Here is the field guide.

The two you will see most and misread most often are "Crawled – currently not indexed" and "Discovered – currently not indexed." They are Google's polite way of saying, respectively, "we looked and decided to pass, at least for now" and "we know it's there but haven't prioritized looking." Neither is an error you can fix with a setting; both are, most often, quality and priority verdicts. A blog with hundreds of thin, similar posts will show a big "Crawled – currently not indexed" number — and the answer isn't a technical toggle, it's a content audit (Chapter 12). This is where technical SEO hands off to content strategy, and reading the report correctly tells you which discipline the problem actually belongs to.

📄 Read the Report

text FIGURE 14.2 — "A small business's indexing report, read correctly" [constructed teaching example] THE QUERY / PAGE Search Console → Page indexing, for a five-location home-services site. WHAT'S THERE Indexed: 47. Not indexed: 96 — "Crawled – currently not indexed": 38 (old thin blog posts) "Excluded by 'noindex' tag": 31 (unexpected — includes 4 service pages!) "Duplicate without user-selected canonical": 19 (tag/archive URLs) "Alternate page with proper canonical tag": 8 (fine) WHAT IT SHOWS Two different stories in one report. The 38 "crawled – not indexed" are a CONTENT problem (thin blog). But 4 real service pages under "noindex" is a TECHNICAL emergency — pages that should rank are actively hidden. WHAT IT DOESN'T It won't tell you WHY those 4 pages have noindex, or which blog posts are worth saving. It flags the symptom; you diagnose the cause (inspect the URLs). THE MOVE Triage by danger, not by number: fix the 4 accidentally-noindexed service pages TODAY; schedule the thin-blog audit (Ch 12) for later; ignore the 8 "alternate" rows entirely. THE LESSON The biggest number is rarely the biggest problem. Read for signal conflicts and accidental exclusions first — a hidden money page beats a hundred harmless ones.

That figure teaches the reading skill that matters: triage by danger, not by count. The scariest line in an indexing report is almost never the largest one. It is the small line that reveals a page you want ranking being accidentally excluded — a noindex on a service page, a valuable URL returning 404, a canonical pointing the wrong way. Those are the invisible catastrophes this chapter exists to prevent.

The companion tool is URL Inspection — type any URL of yours into the search bar at the top of Search Console, and Google tells you, for that specific page: is it on Google? What's the coverage status? What's the Google-selected canonical versus your declared one? When was it last crawled? You can even "Test live URL" to see how Google fetches and renders it right now. When a single important page misbehaves, this is your first stop; when you want the site-wide picture, the Page indexing report is.

🔄 Check Your Understanding Your Page indexing report shows 1,900 URLs indexed, but your site has only about 200 real pages. What is this a symptom of, what's the term for it, and where would you look for the cause?

Answer

This is index bloat — far more URLs indexed than you have real pages. The usual causes are URL variants and low-value auto-generated URLs getting indexed: tracking-parameter versions, tag/category/archive pages, faceted-navigation combinations, session IDs, paginated comment URLs. Look at which URLs are indexed (the report lets you inspect them) to find the pattern, then consolidate with canonicals (§14.3) or apply noindex (§14.4) to the junk. Index bloat dilutes your site's overall quality signal and wastes crawl — the opposite of a clean, focused index.

That term deserves its own definition, because it is the shadow that falls across this whole chapter. Index bloat is the condition of having many low-value, thin, or duplicate URLs sitting in Google's index — parameter URLs, thin archives, near-duplicates, auto-generated pages — so that your indexed footprint is far larger, and far junkier, than your set of genuinely valuable pages. It is harmful in two ways: it can drag down how Google assesses your site's overall quality (a lot of thin pages is itself a signal), and on large sites it wastes crawl budget. The cure is everything you've learned: noindex the pages that shouldn't rank, canonicalize the duplicates, let the genuinely-dead ones 404, and keep your sitemap to only clean, canonical, indexable URLs. A tight index of pages that all deserve to be there beats a bloated one every time. Search Console's indexing reports can show you, honestly and for free, exactly which of your pages Google keeps and why; they cannot fix anything themselves — they are the diagnosis, and the treatment is the rest of this chapter and this book.

📈 The Strategy File

Time to run the first technical audit on Rivertown Home Services. Recall the frozen facts: the Delgados' family company — founded by Ray Delgado in 1984, now run by his children Marisa and Tony — five locations (Rivertown HQ, Cedar Hills, Northgate, Westbrook, Millhaven), a WordPress site, about 8,000 organic visits a month that are almost all branded. Chapter 1 mapped Rivertown's symptoms to pipeline stages. Now we inspect the crawl-and-index layer specifically — sitemap, canonicals, and the hunt for an accidental noindex — the exact increment this chapter owns. We are still diagnosing, not yet rebuilding: architecture is Chapter 15, mobile is Chapter 17, schema is Chapter 18, local is Chapter 25. Today we answer one question for Rivertown: can Google see the pages that matter?

FIGURE 14.3 — "Rivertown's crawl-and-index audit"                    [the Strategy File]
  CHECK                          WHAT WE FOUND                        VERDICT / WHERE IT GOES
  robots.txt                     WordPress default; blocks /wp-admin/  OK — but confirm it does NOT
                                  and internal search. No Disallow: /.  block /wp-content/ (would harm
                                                                        rendering). Add Sitemap: line.
  XML sitemap                    Yoast generates sitemap_index.xml,     Submit it in GSC. Prune junk
                                  BUT it includes 40+ empty tag         later (index-bloat cleanup).
                                  archives and old campaign URLs.
  Accidental noindex             🚨 Settings → Reading "Discourage      EMERGENCY. Uncheck it. This
                                  search engines" box is CHECKED on      alone may be why non-brand
                                  the Millhaven sub-site; 4 service      pages don't rank. Fix TODAY.
                                  pages carry a leftover noindex.
  Canonicals                     Five near-duplicate location pages     Real issue — but the FIX is
                                  ("AC Repair in [City]") mostly self-   content differentiation
                                  canonical, so Google indexes all five  (Ch 15/25), not a canonical
                                  but sees them as near-duplicates.      trick. Note and defer.
  Status codes                   ~12 old blog URLs 404; 3 valuable      404 the dead ones (fine);
                                  ones (linked from Google Business      301-redirect the 3 linked
                                  Profile) also 404.                     URLs. Watch for soft 404s.
  Index bloat                    site: shows ~1,400 indexed vs. ~120     Confirmed bloat from tag
                                  real pages.                            archives + parameters; noindex
                                                                        archives in Yoast.

Read what this audit just did. "Rivertown's website underperforms" was a vague complaint; now it is a triaged, stage-labeled punch list where the most dangerous item — a live noindex hiding real service pages, the classic WordPress "Discourage search engines" checkbox left on — jumps to the front, ahead of a dozen cosmetically scarier but harmless issues. That single accidental noindex could plausibly explain a chunk of why Rivertown is invisible for non-branded searches: you cannot rank a page you have told Google to forget. Notice, too, the discipline of not getting ahead: we found the five near-duplicate location pages, but the fix is real, differentiated local content (Chapters 15 and 25), not a canonical sleight of hand — canonicalizing them away would hide four cities we want to win. Technical SEO tells us the pages can be seen; content and local SEO will make them worth seeing.

Your Strategy-File task for this chapter (Appendix C's worksheet): run the crawl-and-index audit on your own site. (1) Read your robots.txt — anything blocked by accident? (2) Find your sitemap — is it clean? (3) Hunt the accidental noindex: on WordPress, check Settings → Reading for the "Discourage search engines" box, and inspect a few key pages' source for <meta name="robots" content="noindex">. (4) In Search Console, open Page indexing and write down the one or two most dangerous lines — not the biggest, the most dangerous. You are building the technical layer of the strategy that assembles at Chapter 40. Diagnose; don't yet treat.

Conclusion

We set out to answer the question that precedes all others in SEO: can Google actually see this page? And we assembled the control panel that answers it. robots.txt steers crawling — and backfires the moment you ask it to control indexing. Sitemaps aid discovery — and guarantee nothing beyond it. Canonical tags consolidate duplicates onto the URL you choose — as a hint Google can overrule when your other signals disagree. Meta robots and X-Robots-Tag keep pages out of the index cleanly — if you let Google crawl them first. HTTP status codes tell the crawler exactly how to treat a URL — keep it, move it, forget it, or wait. Crawl budget is real for a few large sites and a distraction for everyone else. And Search Console's indexing reports are the honest, free mirror that tells you, page by page, whether all of it is working.

The through-line is the mandatory myth, and it deserves the last word: you cannot noindex a page in robots.txt — because blocking the crawl is exactly what stops Google from ever seeing the noindex. Get that one distinction — crawling is not indexing, and the tools for each are different — and most technical SEO stops being mysterious. It becomes a small set of controls, each with a precise job, that you use to make your site legible to Google rather than accidentally invisible to it. That is not a trick. It is cooperation: telling the search engine the truth about your site so it can do the thing it wants to do, which is show good pages to the people searching for them.

What we did not settle: the audit found that Rivertown's five location pages are near-duplicates and that its architecture starves pages of internal links — real problems whose fixes are not technical toggles but structure and content. That is the natural bridge to the next chapter. If Chapter 14 made sure Google can crawl and index your pages, Chapter 15 asks the deeper question of how the organization of those pages — flat versus deep, URL structure, navigation, and above all internal linking — determines how discovery works and how authority flows through your site. It is where crawlability becomes architecture, and where the most underused lever in all of SEO is waiting.

→ Continue to Chapter 15: Site Architecture.

Key Terms

• robots.txt — a plain-text file at a site's root (following RFC 9309) that tells crawlers which URLs they may and may not fetch; controls crawling, not indexing, and is publicly readable.
• XML sitemap — a machine-readable file listing the URLs you want Google to discover, optionally with a <lastmod> date; a discovery aid, not a guarantee of crawling, indexing, or ranking.
• canonical tag — a <link rel="canonical"> instruction naming the preferred URL among duplicates so Google consolidates signals on it; a strong hint Google can override, not a command.
• duplicate content — substantially identical content reachable at more than one URL; usually not a penalty, but it splits signals and can let Google index a version you didn't choose.
• meta robots — an HTML <head> tag giving page-level indexing instructions to search engines (chiefly noindex and nofollow).
• noindex — an instruction (via meta robots or X-Robots-Tag) telling Google to keep a page out of the index; requires that Google be allowed to crawl the page to read it.
• nofollow — as a page-level meta robots value, an instruction not to follow the links on a page; distinct from the link-level rel="nofollow" attribute covered in Chapter 22.
• X-Robots-Tag — the same directives as meta robots delivered via an HTTP response header, used to control non-HTML files (PDFs, images) that have no HTML <head>.
• HTTP status code — the three-digit code a server returns for each request (200, 301, 302, 304, 404, 410, 500, 503, and others); read by Google as an instruction about how to treat the URL.
• soft 404 — a page that returns a 200 OK status but has "not found" or empty content, misleading Google into thinking a real page exists; a defect to fix.
• index bloat — the condition of having many low-value, thin, or duplicate URLs in Google's index, diluting a site's quality signal and wasting crawl; cured by noindex, canonicals, and clean sitemaps.

Spaced Review

Retrieval practice mixing this chapter with Chapter 1. Try each before revealing the answer.

1. Explain, in two sentences, why you cannot use robots.txt to remove a page from Google's search results — and what you should use instead.
2. Match each goal to the correct control: (a) "keep this thank-you page out of search," (b) "this URL moved permanently," (c) "these tracking-parameter URLs are duplicates of one page," (d) "don't crawl this junk folder at all."
3. (Chapter 1 recall) Name the five stages of the pipeline in order, and say which two stages this chapter's tools primarily govern.
4. (Chapter 1 recall) A site owner says "optimize my crawl budget." For what kind of site is that usually a waste of effort, and roughly what URL count is the threshold below which most sites needn't worry?
5. In a Search Console indexing report, why is "Excluded by 'noindex' tag" on four of your service pages more urgent than "Crawled – currently not indexed" on forty old blog posts?