Chapter 14 — Quiz

Twenty-six self-check questions: eighteen multiple choice, then eight short answer. Answer them before you open the key at the bottom. This quiz tests the one distinction the whole chapter turns on — crawling is not indexing — and the tools that control each.

Part 1 — Multiple choice

Q1. robots.txt primarily controls: - A. Which pages Google indexes - B. Which URLs a crawler may fetch - C. Your rankings for competitive queries - D. How fast your pages load

Q2. Which statement is correct? - A. A URL blocked by robots.txt can never appear in Google - B. A URL blocked by robots.txt can still be indexed (as a bare link) if other pages link to it - C. robots.txt is the best way to de-index a page - D. robots.txt hides confidential files from the public

Q3. To reliably keep a page out of Google's index, you should: - A. Add Disallow for it in robots.txt - B. Add a Noindex: line to robots.txt - C. Allow crawling and add a noindex meta robots tag on the page - D. Delete it from your sitemap

Q4. Why does blocking a page in robots.txt and adding a noindex fail to remove it? - A. noindex only works on images - B. Google must crawl the page to read the noindex, but robots.txt blocks the crawl - C. You can only use one directive per page - D. It works fine; there is no conflict

Q5. An XML sitemap is best described as: - A. A guarantee that listed pages will be indexed - B. A ranking factor - C. A discovery aid that helps Google find your URLs - D. A file that blocks crawlers

Q6. Google has stated that in a sitemap it ignores: - A. The <loc> (URL) field - B. <changefreq> and <priority> - C. The whole file - D. <lastmod> always

Q7. The canonical tag is: - A. A strict command Google must obey - B. A strong hint Google can override if other signals disagree - C. A way to block crawling - D. A redirect

Q8. "Duplicate content" on a normal site usually results in: - A. A manual penalty - B. Deindexing of the whole site - C. Google picking one version and consolidating — no penalty - D. A ranking boost

Q9. Which HTTP status code says "this URL moved permanently; pass signals to the new one"? - A. 302 - B. 301 - C. 404 - D. 503

Q10. During planned maintenance, the correct status code to return is: - A. 200 - B. 404 - C. 500 - D. 503

Q11. A soft 404 is: - A. A page correctly returning 404 - B. A page that returns 200 OK but has "not found" or empty content - C. A permanent 410 - D. A redirect loop

Q12. Google's position on ordinary 404s is that they: - A. Severely damage your whole site's rankings - B. Are normal and do not harm the rest of your site - C. Must all be redirected, without exception - D. Should be returned as 200 instead

Q13. For most sites, "crawl-budget optimization" is: - A. Essential from day one - B. A non-issue; a distraction from real work - C. Required for sites under 1,000 pages - D. A ranking factor

Q14. Roughly what URL count does Google's guidance use as the threshold below which sites generally needn't worry about crawl budget? - A. About 100 - B. About 1,000 - C. About 10,000 - D. About 1,000,000

Q15. "Crawled – currently not indexed" most often means: - A. Google can't access the page - B. Google fetched the page and chose not to index it (often a quality/priority judgment) - C. The page has a noindex tag - D. The page 404s

Q16. The X-Robots-Tag is used chiefly to: - A. Speed up your site - B. Apply noindex/nofollow to non-HTML files (like PDFs) via an HTTP header - C. Block crawlers, like robots.txt - D. Submit a sitemap

Q17. Index bloat refers to: - A. Slow page load from large images - B. Many low-value/duplicate URLs sitting in Google's index - C. Too many backlinks - D. A sitemap that is too large

Q18. In an indexing report, the most urgent line is usually: - A. Whichever has the highest count - B. A small line showing a page you want ranked being excluded (e.g., noindex on a service page) - C. "Alternate page with proper canonical tag" - D. "Page with redirect"

Part 2 — Short answer

Q19. State the single most important sentence in this chapter about robots.txt (what it controls versus what it doesn't).

Q20. A colleague wants to remove a live page from Google. Give the correct method in one sentence, and say why blocking it in robots.txt would fail.

Q21. Explain the difference between a canonical tag, a noindex, and a 301 redirect — one sentence each, naming the job each does.

Q22. List the eight status codes this chapter says matter, and give the one-line meaning of any three.

Q23. Why is putting a URL in your sitemap not enough to get it indexed?

Q24. Give two of the three real costs of unmanaged duplicate content (remembering it is usually not a penalty).

Q25. Name the free report you would open to see, page by page, which of your URLs Google keeps and why — and one status label you would treat as an emergency if it appeared on an important page.

Q26. A 250-page site's consultant recommends a crawl-budget project. In two sentences, explain how you'd decide whether it's needed and what evidence you'd check.

Answer key (try every question first) **Part 1 — Multiple choice** 1. **B** — `robots.txt` controls *crawling* (fetching), not indexing. 2. **B** — Blocked URLs can still be indexed from links, as bare, snippet-less listings. 3. **C** — Allow the crawl and put a `noindex` on the page so Google can read it. 4. **B** — Blocking the crawl blindfolds Google to the `noindex`; the two controls conflict. 5. **C** — A discovery aid. It is not an indexing guarantee and not a ranking factor. 6. **B** — Google ignores `` and ``; it may use an accurate ``. 7. **B** — A strong hint/signal Google can override when other signals disagree. 8. **C** — Google picks one version and consolidates; there is no penalty for ordinary duplication. 9. **B** — 301 Moved Permanently. 10. **D** — 503 Service Unavailable (ideally with a `Retry-After` header). 11. **B** — A `200 OK` response on a page that is really "not found"/empty. 12. **B** — Normal and not harmful to the rest of the site; only *valuable* dead URLs should be redirected. 13. **B** — A non-issue and a distraction for most sites. 14. **C** — About 10,000 URLs (generally, up to a few thousand needn't worry at all). 15. **B** — Google crawled it and declined to index — usually a quality/priority verdict. 16. **B** — Delivers robots directives via HTTP header, for non-HTML files without a ``. 17. **B** — Many low-value/duplicate URLs in the index, diluting quality signals and wasting crawl. 18. **B** — The small line exposing an accidentally-excluded page you want to rank; triage by danger, not count. **Part 2 — Short answer** 19. **`robots.txt` controls crawling, not indexing** — it tells crawlers what they may *fetch*, not what Google may keep in (or out of) search results. 20. Allow crawling and add a `` (or X-Robots-Tag) to the page; blocking it in `robots.txt` fails because Google must crawl the page to *see* the `noindex`, and the block prevents that — so the URL can stay indexed. 21. **Canonical tag:** "these are duplicates — index this one" (consolidate, both URLs may remain reachable). **`noindex`:** "keep this page out of the index entirely." **301 redirect:** "this URL is permanently gone — send everyone (and its signals) to the new URL." 22. 200 (OK/serve), 301 (permanent redirect), 302 (temporary redirect), 304 (not modified — use cache), 404 (not found), 410 (gone, permanently), 500 (server error — back off), 503 (temporarily unavailable — retry later). (Any three one-liners accepted.) 23. Because a sitemap only aids *discovery*; indexing is a separate quality gate, and Google routinely declines to index sitemap URLs it judges thin, duplicate, or low-value. 24. Any two of: (1) Google may index a version you didn't choose; (2) ranking signals get split across the duplicate URLs instead of concentrating on one; (3) crawl effort is wasted re-fetching copies. 25. The **Page indexing** report in Google Search Console; an emergency label on an important page would be **"Excluded by 'noindex' tag"** (or "Indexed, though blocked by robots.txt," or a 404/soft 404 on a money page). 26. Check the site's size and Search Console's **Crawl stats**: if it's under ~10,000 URLs on a fast server and Google is already crawling it fully, crawl budget isn't the bottleneck and the project is unnecessary; the evidence is the URL count plus the crawl-requests and response-time figures in Crawl stats.