Affiliate disclosure

Book titles on this page link to Amazon. As an Amazon Associate, DataField.Dev earns from qualifying purchases — at no additional cost to you.

Further Reading: Web Scraping and Automation

Web Scraping Fundamentals

  • Sweigart, A. (2020). Automate the Boring Stuff with Python, 2nd Ed. No Starch Press. Chapters 12-13. (Tier 1) Chapter 12 covers web scraping with Beautiful Soup, and Chapter 13 covers working with Excel, PDFs, and other file formats. The entire book is freely available online. This is the single best companion resource for this chapter's material. https://automatetheboringstuff.com/2e/

  • Mitchell, R. (2018). Web Scraping with Python, 2nd Ed. O'Reilly Media. (Tier 1) The definitive book on web scraping with Python. Covers Beautiful Soup, Selenium, Scrapy, and advanced topics like handling JavaScript-heavy sites, CAPTCHAs, and legal considerations. More depth than you need now, but an excellent reference as your projects grow.

  • Beautiful Soup Documentation. (Tier 1) The official docs are well-written and full of examples. The "Quick Start" section covers everything from this chapter; the full docs cover edge cases you'll encounter in real projects. https://www.crummy.com/software/BeautifulSoup/bs4/doc/

The requests Library

  • Requests Documentation: "Quickstart." (Tier 1) The official guide to Python's most popular HTTP library. Covers GET/POST requests, headers, authentication, sessions, and error handling. You used requests in Chapter 21 — this resource goes deeper. https://docs.python-requests.org/en/latest/user/quickstart/

  • Reitz, K. & Schlusser, C. (2016). The Hitchhiker's Guide to Python. O'Reilly Media. (Tier 2) An opinionated guide to writing great Python. The "Web Scraping" section provides practical advice on choosing tools and structuring scraping projects. https://docs.python-guide.org/

  • Sellars, A. (2018). "Twenty Years of Web Scraping and the Computer Fraud and Abuse Act." Boston University Journal of Science & Technology Law, 24(2). (Tier 2) An accessible legal analysis of how US courts have treated web scraping under the CFAA. Important reading if you're considering scraping for professional or commercial purposes. Available via SSRN.

  • hiQ Labs, Inc. v. LinkedIn Corp., 938 F.3d 985 (9th Cir. 2019). (Tier 2) A landmark US court case establishing that scraping publicly available data does not violate the Computer Fraud and Abuse Act. The Ninth Circuit ruled that LinkedIn could not use the CFAA to prevent hiQ from scraping public profiles. Significant for understanding the legal landscape, but not a license to scrape anything — terms of service and other laws still apply.

  • The robots.txt Standard (RFC 9309). (Tier 1) The official internet standard for the Robots Exclusion Protocol. Short and readable — explains exactly how robots.txt files work, what Disallow, Allow, and Crawl-delay mean, and how crawlers should interpret them. https://www.rfc-editor.org/rfc/rfc9309

Automation and Scheduling

  • Python Documentation: pathlib — Object-oriented filesystem paths. (Tier 1) Essential for file automation tasks. You've used pathlib since Chapter 10 — this reference covers every method available on Path objects. https://docs.python.org/3/library/pathlib.html

  • Python Documentation: shutil — High-level file operations. (Tier 1) Covers file copying, moving, and archiving. Critical for the file organization automation patterns in Section 24.6. https://docs.python.org/3/library/shutil.html

  • schedule library documentation. (Tier 1) A lightweight Python library for in-process job scheduling. Simpler than cron for Python-specific tasks, though it requires the script to stay running. https://schedule.readthedocs.io/

  • Crontab Guru. (Tier 1) An interactive tool for building and testing cron expressions. Paste a cron expression and see in plain English when it will run. Invaluable when you're learning cron syntax. https://crontab.guru/

Advanced Scraping Tools

  • Selenium Documentation. (Tier 2) When a website requires JavaScript to render its content, Beautiful Soup alone isn't enough. Selenium controls a real web browser programmatically. The official docs cover setup, finding elements, waiting for page loads, and handling dynamic content. https://www.selenium.dev/documentation/

  • Playwright Documentation. (Tier 2) A modern alternative to Selenium by Microsoft. Faster, more reliable, and with better async support. Worth exploring after you're comfortable with basic scraping. https://playwright.dev/python/

  • Scrapy Documentation. (Tier 2) An industrial-strength web scraping framework for Python. Overkill for learning, but invaluable for large-scale scraping projects with hundreds or thousands of pages. Built-in support for concurrency, pipelines, and middleware. https://docs.scrapy.org/

Practice Sites for Scraping

  • Quotes to Scrape — http://quotes.toscrape.com (used in this chapter)
  • Books to Scrape — http://books.toscrape.com
  • HTTPBin — https://httpbin.org (for testing HTTP requests)
  • The Scrapinghub Blog's list of practice sites — search "web scraping sandbox" for additional options

For the Curious

  • Fielding, R. T. (2000). "Architectural Styles and the Design of Network-Based Software Architectures." Doctoral dissertation, UC Irvine. (Tier 3) The dissertation that defined REST — the architectural style behind the web APIs you used in Chapter 21 and the web pages you're scraping in this chapter. Dense and academic, but foundational. Chapter 5 is the most relevant.

  • Berners-Lee, T. (1999). Weaving the Web. HarperCollins. (Tier 2) The inventor of the World Wide Web tells the story of how HTML, HTTP, and URLs were designed. Understanding why the web works the way it does makes you a better scraper and a better web developer.