Chapter 14 — Key Takeaways
A one-page reference. If you remember nothing else, remember that crawling is not indexing — and that the tools for each are different.
The one idea
Before you ask why a page doesn't rank, ask whether Google can even see it. robots.txt controls
crawling; noindex controls indexing; and you cannot noindex a page in robots.txt, because blocking
the crawl is exactly what stops Google from seeing the noindex. Technical SEO is making your site legible
to Google, not accidentally invisible to it.
The four controls (don't confuse them)
| Goal | Tool | Key limit |
|---|---|---|
| Don't crawl this | Disallow in robots.txt |
Doesn't de-index; blocked URLs can still be indexed from links |
| Don't index this | noindex (meta robots / X-Robots-Tag) |
Google must be allowed to crawl the page to read it |
| These are duplicates — index one | canonical tag | A hint, not a command; Google can override it |
| This URL moved permanently | 301 redirect | For real moves; not for consolidating live duplicates |
The status codes that matter
| Code | Means | Tells Google |
|---|---|---|
| 200 | OK | Serve it |
| 301 | Moved permanently | Pass signals to the new URL |
| 302 | Temporary | Keep the old URL indexed |
| 304 | Not modified | Use your cached copy |
| 404 | Not found | Drop it (normal; no penalty) |
| 410 | Gone | Drop it, a bit faster |
| 500 | Server error | Back off; can drop pages if it persists |
| 503 | Unavailable | Retry later — the right code for maintenance |
Rules of thumb
- To de-index a page: allow the crawl +
noindex. Neverrobots.txtalone. Block last, after it's out. - A sitemap aids discovery — it does not force indexing and is not a ranking factor. Keep it clean (only canonical, indexable, 200-status URLs).
- Duplicate content is (almost always) not a penalty. It splits signals and can let Google pick the wrong URL — so canonicalize, don't panic.
- 404s don't hurt your site. Let dead pages 404; only 301-redirect valuable dead URLs; hunt down soft 404s (a 200 on a "not found" page).
- Crawl budget is a non-issue for most sites (under ~10,000 URLs on a decent server). It matters at real scale; it is never a ranking factor. Confirm with Crawl stats and move on.
- Read an indexing report by danger, not by count. A
noindexon one service page beats a hundred harmless "alternate page" rows.
The myths, corrected
- ❌ "Add
noindextorobots.txt." → You can't; it does nothing, and blocking the crawl hides your realnoindex. - ❌ "A sitemap gets pages indexed." → It only helps discovery.
- ❌ "Duplicate content is a penalty." → It's a consolidation/efficiency issue, not a punishment.
- ❌ "404s hurt rankings." → They're normal; only valuable dead URLs need redirecting.
- ❌ "Everyone must optimize crawl budget." → Most sites needn't; it's for the large-site minority.
What you can do today
Run the four-step crawl-and-index audit: (1) read your /robots.txt; (2) find and skim your sitemap;
(3) hunt for an accidental noindex (on WordPress, check Settings → Reading); (4) open Search Console's
Page indexing report and write down the one or two most dangerous lines. That triaged list is the
technical layer of your Strategy File.
Where this goes next
→ Chapter 15 turns crawlability into architecture: how flat-versus-deep structure, URLs, navigation, and above all internal linking determine how Google discovers your pages and how authority flows through your site.