Chapter 14 — Exercises

Work these in order; they climb from recall to real diagnosis. You will need a live web browser and, ideally, a website you can inspect (your own, your employer's, or a client's) plus Google Search Console access if you have it. Every tool used here is free. Items marked with a dagger (†) have full worked solutions in Appendix — Answers to Selected Exercises; odd-numbered items are worked there too. Attempt every item before you look.

The habit this chapter trains: before asking "why doesn't this rank?", ask "can Google even see this?" Crawlability and indexability come first.

A. Recall and vocabulary

Exercise 14.1 — In one sentence, state what robots.txt controls and what it does not control. Then finish this sentence without looking back: "A URL blocked by robots.txt can still be ______."

Exercise 14.2 — Define duplicate content in your own words, and list four ordinary, non-malicious ways it gets created on a normal website.

Exercise 14.3 † — Explain the difference between a 301 and a 302 status code, and why using a 302 when you mean a 301 is a mistake.

Exercise 14.4 — What is an XML sitemap for? Name two things it does and two things it does not do.

Exercise 14.5 — Distinguish a 404 from a soft 404. Which one is a genuine defect, and why?

Exercise 14.6 † — Define index bloat and name three common sources of it. Why is a smaller, cleaner index usually better than a large one?

B. Applied reasoning

Exercise 14.7 — A developer proposes keeping the company's staging site out of Google by adding Disallow: / to the staging site's robots.txt. A marketer proposes a site-wide noindex instead. For a staging site that must never appear in search, which is safer, and what is the risk of the robots.txt approach if the file is ever copied to production?

Exercise 14.8 † — Explain, for a non-technical business owner, why "just block it in robots.txt" is the wrong way to remove an existing page from Google. Use no jargon the owner wouldn't know, and end with the one-sentence rule they should remember.

Exercise 14.9 — You have the same product page reachable at six URLs (http/https, www/non-www, and with two tracking parameters). Describe how you would consolidate them so Google indexes one clean URL, and name the specific tool for each part of the fix.

Exercise 14.10 — Google's documentation says the canonical tag is a "signal," not a "directive." Explain what that distinction means in practice, and describe one situation where Google would override the canonical you declared.

Exercise 14.11 † — Your company is planning two hours of scheduled maintenance during which the site will be offline. What HTTP status code should the server return during that window, what header should accompany it, and what would go wrong if the site returned 200 OK with a "we'll be back soon" page instead?

Exercise 14.12 — Explain why "put the page in the sitemap" will not get a thin, duplicate page indexed, tying your answer to the idea (from Chapter 1) that indexing is a quality gate.

C. Diagnose the problem

For each scenario, name the most likely cause, the stage of the pipeline it sits at, and the single check or fix you would reach for first. Some have more than one defensible answer — say why.

Exercise 14.13 † — A page shows "Indexed, though blocked by robots.txt" in Search Console. The owner wanted this page out of Google and is confused that it's in. Explain exactly what happened and how to actually remove it.

Exercise 14.14 — After a redesign, a whole section of the site vanished from Google over two weeks. The pages load fine, return 200 OK, and are in the sitemap. Name the most likely single cause and the two-minute check that would confirm it.

Exercise 14.15 — A blog with 400 posts shows 250 of them as "Crawled – currently not indexed." The owner asks whether they should build links to those posts to "push them into the index." Diagnose the real problem and explain why link building is the wrong first move.

Exercise 14.16 † — A site:yourdomain search returns roughly 9,000 results for a site with about 300 real pages. Name the condition, list the two most likely URL types inflating the count, and give the two tools you'd use to fix it.

Exercise 14.17 — Three valuable old URLs that are linked from the company's Google Business Profile now return 404 after a URL restructure. Is a 404 the right response for these, and if not, what is — and why does it matter that they are linked from elsewhere?

Exercise 14.18 — A single important service page is missing from Google. In URL Inspection, its "Google-selected canonical" is a different page than the URL itself. What does that tell you, and what would you check to fix it?

D. On your own site (use a free tool)

Exercise 14.19 † — Visit yourdomain.example/robots.txt. Record what it disallows. Flag anything that looks risky (a Disallow: /, a blocked resource folder like /wp-content/, a blocked section you actually want crawled). Write one sentence on whether the file is doing its job.

Exercise 14.20 — Find your XML sitemap (try /sitemap.xml, /sitemap_index.xml, and the Sitemap: line in robots.txt). Open it and answer: does it contain any URLs you would not want indexed (tag archives, old campaigns, parameter URLs)? Note two you would remove.

Exercise 14.21 † — Hunt for an accidental noindex. If you're on WordPress, check Settings → Reading for the "Discourage search engines from indexing this site" checkbox. On any platform, open two important pages, use "View Page Source," and search for noindex. Record what you find. (If you find one on a page you want ranked, you have just found the highest-value fix in this chapter.)

Exercise 14.22 — In Search Console, open Settings → Crawl stats. Record your average server response time and total crawl requests over the last 90 days. Is Google crawling far more URLs than you have real pages? Write one sentence interpreting whether crawl budget is (or isn't) a concern for your site.

E. Read the report

Exercise 14.23 † — Interpret this indexing report and triage it.

FIGURE E14.1 — "An indexing report to triage"                 [constructed teaching example]
  Indexed: 62
  Not indexed: 210 —
    "Crawled – currently not indexed": 120
    "Duplicate without user-selected canonical": 55
    "Excluded by 'noindex' tag": 28  (includes the /pricing/ page)
    "Blocked by robots.txt": 7

Which single line should alarm you most, and why? Rank the four not-indexed reasons from most to least urgent for a business that sells through its website, and justify the order.

Exercise 14.24 — You are handed this robots.txt from a live site that "isn't getting any organic traffic." Identify the two lines that are most likely responsible and explain what each is doing.

User-agent: *
Disallow: /
Disallow: /wp-content/
Sitemap: https://example.com/sitemap.xml

F. Myths, ethics, and judgment

Exercise 14.25 † — Spot the myth. A popular thread claims: "Duplicate content triggers a Google penalty, so never let two URLs show similar text." Identify what is false, state what actually happens when Google finds duplicates, and name the one case where duplication is penalized.

Exercise 14.26 — Spot the myth. A consultant tells a 250-page local business it urgently needs a "\$3,000 crawl-budget optimization." Using the chapter (and Google's own guidance), explain why this is almost certainly unnecessary, and what evidence you'd show the owner to prove it.

Exercise 14.27 — An SEO wants to hide a folder of confidential client PDFs from Google by adding a Disallow rule in robots.txt. Explain why this is both ineffective and a security risk, and state what should be done instead.

G. The Strategy File (Rivertown Home Services)

Exercise 14.28 † — Rivertown's Chapter 14 audit (Figure 14.3) found the WordPress "Discourage search engines" checkbox enabled on the Millhaven sub-site and a leftover noindex on four service pages. Write the three-step remediation you'd hand the Delgados' web person, in priority order, and explain why this jumps ahead of the near-duplicate location pages on the punch list.

Exercise 14.29 — Rivertown's audit found ~1,400 URLs indexed versus ~120 real pages, driven by empty tag archives and parameter URLs. Propose the correct fix for each source (archives vs. parameter URLs), naming the specific control, and explain why you would not simply Disallow them all in robots.txt.

Exercise 14.30 — Do Rivertown's Chapter 14 Strategy-File task on your own (or a client's) site: run the four-part crawl-and-index audit (robots.txt, sitemap, accidental-noindex hunt, Page indexing report) and produce a triaged list of at least four findings, each tagged emergency / real issue / defer / harmless. Solve nothing yet — the deliverable is a triaged technical punch list, the artifact Rivertown's rebuild adds this chapter.

Where these go next. Keep your answers to 14.19–14.22, 14.28, and 14.30 — they are the technical layer of your running Strategy File (Appendix C's worksheet), and they feed directly into the full technical audit you assemble in Chapter 38 and the capstone in Chapter 40.