Further Reading: Web Scraping and Automation

Web Scraping Fundamentals

• Sweigart, A. (2025). Automate the Boring Stuff with Python, 3rd Ed. No Starch Press. Chapters 11, 13, and 19. (Tier 1) The book's Chapter 13 covers web scraping with requests and Beautiful Soup, Chapter 11 covers organizing files, and Chapter 19 covers keeping time, scheduling tasks, and launching programs: together, a close match for Sections 24.3–24.7. The entire book is freely available online. This is the single best companion resource for this chapter's material. (In the 2nd edition, the same topics are Chapters 12, 10, and 17.) https://automatetheboringstuff.com/3e/

• Mitchell, R. (2024). Web Scraping with Python, 3rd Ed. O'Reilly Media. (Tier 1) The definitive book on web scraping with Python. Covers Beautiful Soup, Selenium, Scrapy, and advanced topics like handling JavaScript-heavy sites, reading text from images, and the legalities and ethics of scraping. More depth than you need now, but an excellent reference as your projects grow.

• Beautiful Soup Documentation. (Tier 1) The official docs are well-written and full of examples. The "Quick Start" section covers everything from this chapter; the full docs cover edge cases you'll encounter in real projects. https://www.crummy.com/software/BeautifulSoup/bs4/doc/

The requests Library

• Requests Documentation: "Quickstart." (Tier 1) The official guide to Python's most popular HTTP library. Covers GET/POST requests, headers, authentication, sessions, and error handling. You used requests in Chapter 21 — this resource goes deeper. https://requests.readthedocs.io/en/latest/user/quickstart/

• Reitz, K. & Schlusser, C. (2016). The Hitchhiker's Guide to Python. O'Reilly Media. (Tier 2) An opinionated guide to writing great Python. Its short "HTML Scraping" page shows the same fetch-then-parse pattern with a different parser, lxml, which is a useful way to see that Beautiful Soup isn't the only option. https://docs.python-guide.org/scenarios/scrape/

Ethical and Legal Considerations

• Sellars, A. (2018). "Twenty Years of Web Scraping and the Computer Fraud and Abuse Act." Boston University Journal of Science & Technology Law, 24, 372. (Tier 2) An accessible legal analysis of how US courts have treated web scraping under the CFAA, written just before the hiQ rulings below. Important reading if you're considering scraping for professional or commercial purposes. Free on SSRN. https://papers.ssrn.com/abstract=3221625

• hiQ Labs, Inc. v. LinkedIn Corp., 938 F.3d 985 (9th Cir. 2019); reaffirmed 31 F.4th 1180 (9th Cir. 2022). (Tier 2) The best-known US scraping case (Section 24.5.3). The Ninth Circuit held that scraping publicly viewable LinkedIn profiles likely wasn't "access without authorization" under the Computer Fraud and Abuse Act. The rest of the story matters just as much: in late 2022 the district court found that hiQ had breached LinkedIn's User Agreement, and hiQ settled, agreeing to stop scraping and delete the data. Read it as "the CFAA isn't the only rule," not as a license to scrape anything public.

• The robots.txt Standard (RFC 9309). (Tier 1) The official internet standard for the Robots Exclusion Protocol, published in 2022. Short and readable — explains exactly how robots.txt files work, what User-agent, Allow, and Disallow mean, and what a crawler should do when the file is missing or unreachable. Notice what it doesn't define: Crawl-delay, which is a widely used extension rather than part of the standard. https://www.rfc-editor.org/rfc/rfc9309

Automation and Scheduling

• Python Documentation: pathlib — Object-oriented filesystem paths. (Tier 1) Essential for file automation tasks. You've used pathlib since Chapter 10 — this reference covers every method available on Path objects. https://docs.python.org/3/library/pathlib.html

• Python Documentation: shutil — High-level file operations. (Tier 1) Covers file copying, moving, and archiving. Critical for the file organization automation patterns in Section 24.6. https://docs.python.org/3/library/shutil.html

• schedule library documentation. (Tier 1) A lightweight Python library for in-process job scheduling. Simpler than cron for Python-specific tasks, though it requires the script to stay running. https://schedule.readthedocs.io/

• Crontab Guru. (Tier 1) An interactive tool for building and testing cron expressions. Paste a cron expression and see in plain English when it will run. Invaluable when you're learning cron syntax. https://crontab.guru/

Advanced Scraping Tools

• Selenium Documentation. (Tier 2) When a website requires JavaScript to render its content, Beautiful Soup alone isn't enough. Selenium controls a real web browser programmatically. The official docs cover setup, finding elements, waiting for page loads, and handling dynamic content. https://www.selenium.dev/documentation/

• Playwright Documentation. (Tier 2) A modern alternative to Selenium by Microsoft. Faster, more reliable, and with better async support. Worth exploring after you're comfortable with basic scraping. https://playwright.dev/python/

• Scrapy Documentation. (Tier 2) An industrial-strength web scraping framework for Python. Overkill for learning, but invaluable for large-scale scraping projects with hundreds or thousands of pages. Built-in support for concurrency, pipelines, and middleware. https://docs.scrapy.org/

Practice Sites for Scraping

• Quotes to Scrape — http://quotes.toscrape.com (used in this chapter)
• Books to Scrape — http://books.toscrape.com
• HTTPBin — https://httpbin.org (for testing HTTP requests)
• Web Scraping Sandbox — https://toscrape.com (the home page for both sites above; it lists extra versions of the quotes site built around one challenge each, such as JavaScript-generated content at /js/, infinite scrolling at /scroll, and a login form at /login)

For the Curious

• Fielding, R. T. (2000). "Architectural Styles and the Design of Network-Based Software Architectures." Doctoral dissertation, UC Irvine. (Tier 3) The dissertation that defined REST — the architectural style behind the web APIs you used in Chapter 21 and the web pages you're scraping in this chapter. Dense and academic, but foundational. The dissertation's Chapter 5, "Representational State Transfer (REST)," is the most relevant. https://ics.uci.edu/~fielding/pubs/dissertation/top.htm

• Berners-Lee, T. (1999). Weaving the Web. HarperCollins. (Tier 2) The inventor of the World Wide Web tells the story of how HTML, HTTP, and URLs were designed. Understanding why the web works the way it does makes you a better scraper and a better web developer.