Chapter 24: Web Scraping and Automation

"...use Python to write programs that do in minutes what would take you hours to do by hand." — From the description of Al Sweigart's Automate the Boring Stuff with Python

Chapter Overview

Every Friday afternoon, Elena Vasquez sat down at her desk at Harbor Community Services and began her four-hour ritual. Open three browser tabs. Navigate to three county data portals. Click "Export." Wait. Download. Copy-paste each dataset into a master spreadsheet. Check for obvious errors — sometimes missing them, like the column swap that once led to wrong numbers at a board meeting. Write formulas. Format the report. Email it to twelve stakeholders. Four hours, every single week.

You first met Elena in Chapter 1. You've watched her story evolve — her first loop in Chapter 5, file I/O in Chapter 10, data processing in Chapter 21, regex validation in Chapter 22. Now, in Chapter 24, her story reaches its payoff. Elena's four-hour Friday becomes a 30-second script execution, scheduled to run at 6 AM every Friday before she even gets to the office.

This chapter teaches you two related skills: web scraping (extracting data from web pages programmatically) and automation (making the computer do repetitive work without your involvement). Web scraping lets your programs read web pages the way you do — but faster and without getting bored. Automation turns any repetitive workflow into a script that runs on command or on schedule.

In this chapter, you will learn to: - Parse HTML documents and extract data using Beautiful Soup - Use CSS selectors and Beautiful Soup methods to locate specific elements - Build a complete web scraper with error handling, rate limiting, and polite headers - Apply ethical scraping practices: robots.txt, rate limits, Terms of Service - Automate file management, report generation, and batch processing - Schedule scripts with cron, Task Scheduler, and the schedule library - Design a complete fetch-parse-transform-output pipeline

🏃 Fast Track: If you're familiar with HTML from web development, skim 24.2, then focus on 24.3 (Beautiful Soup API) and 24.5 (ethics). If you just want the pipeline pattern, jump to 24.8.

🔬 Deep Dive: The case studies explore Elena's fully automated report pipeline and a detailed ethical analysis of five scraping scenarios along a spectrum from clearly ethical to clearly problematic.

Spaced Review: This chapter builds directly on Chapter 21 (HTTP requests with requests, JSON/CSV processing) and Chapter 22 (regex for text extraction). You'll also use file I/O from Chapter 10 and module structure from Chapter 12. If requests.get() or re.findall() feel rusty, revisit those chapters before continuing.

24.1 Why Automate?

Here's a question worth sitting with: how much of your work is genuinely creative, and how much is mechanical repetition?

Elena's weekly report was valuable — the executive director relied on it. But the process of creating it was pure drudgery. The same clicks, the same copy-pastes, the same formulas, the same formatting, every single week. Nothing required human judgment except catching errors — and humans are terrible at catching errors in repetitive work. (Ask Elena about the column-swap incident.)

Consider the math. Elena's report took 4 hours per week. Over a working year of about 50 weeks, that's 200 hours — five full 40-hour work weeks. Her Python script takes 30 seconds. Even after subtracting the 8 hours she spent writing and debugging the script (and the 25 or so minutes the script itself runs across the year), she saves about 191.5 hours in the first year alone. Every subsequent year, she saves the full 200 hours.

But the real multiplier isn't just time. It's reliability. A script doesn't get tired on Friday afternoon. It doesn't accidentally swap two columns. It doesn't forget to check for negative numbers in the data. It runs the same validation checks every time, catches anomalies that a human would miss, and produces output in exactly the format stakeholders expect.

Here's what Elena's pipeline does now versus what she used to do manually:

🚪 Threshold Concept: Automation as a Multiplier

Before this chapter: "Programming is about solving one problem at a time."

After this chapter: "Programming is about building machines that solve the same problem forever — while I move on to the next one."

This shift changes how you look at every repetitive task. When you catch yourself doing the same thing for the third time, a voice in your head will ask: "Could I automate this?" Often the answer is yes. Renaming 500 files? Script. Checking a website for updates? Script. Generating a weekly report from three data sources? Script. Sending reminder emails to your study group? Script.

More importantly, automated pipelines are reliable. They don't have bad days. They don't forget steps. They produce consistent, auditable results every run. In Elena's case, the column-swap error that embarrassed her at a board meeting (Chapter 1) is now structurally impossible — the script validates data before any calculations happen.

Once you see the world through the automation lens, you can't unsee it. Every manual process becomes a potential script. Every repetitive task becomes an opportunity to build something permanent. The cost of automation is fixed; the benefit compounds.

24.2 HTML Basics for Scraping

Web scraping means extracting data from web pages. Web pages are written in HTML (HyperText Markup Language) — a language of nested tags that tells your browser what to display. You don't need to become an HTML expert, but you do need to understand enough to tell your scraper where to find the data.

24.2.1 Tags, Attributes, and Content

HTML is built from elements, each defined by a tag:

<p class="description">Learn web scraping with Beautiful Soup.</p>

Let's break this down:

Tags nest inside each other, forming a tree structure:

<div class="event" data-date="2025-09-15">
  <h2>Python Workshop</h2>
  <p class="description">Learn web scraping with Beautiful Soup.</p>
  <span class="location">Room 204, CS Building</span>
</div>

Here, the <div> is the parent element. The <h2>, <p>, and <span> inside it are children. This parent-child relationship is how HTML organizes information — and how your scraper will navigate to find the data you need.

24.2.2 The Tags You'll See Most Often

24.2.3 Classes and IDs

Two attributes are especially important for scraping:

• class: A label shared by multiple elements. On a product page, every product might be inside a <div class="product">. You use classes to find all elements of a certain type.
• id: A unique identifier. Only one element on the page should have a given ID. You use IDs to find one specific element.

24.2.4 HTML as a Tree

HTML's nested structure forms a tree:

html
├── head
│   └── title
└── body
    ├── div#results
    │   ├── div.product
    │   │   ├── h2.product-name  ->  "Widget A"
    │   │   └── span.price       ->  "$12.99"
    │   └── div.product
    │       ├── h2.product-name  ->  "Widget B"
    │       └── span.price       ->  "$8.49"

Beautiful Soup turns this tree into Python objects you can navigate — by tag name, by class, by ID, or by position in the tree.

💡 Tip: Most web browsers let you inspect a page's HTML structure. Right-click on any element and choose "Inspect" (Chrome, Edge) or "Inspect Element" (Firefox, Safari). The developer tools panel shows you the exact HTML tags, classes, and IDs — invaluable for figuring out what to target in your scraper.

🔗 Connection — Spaced Review (Ch 10): In Chapter 10, you learned to read files line by line and parse their structure. HTML parsing is conceptually similar — you're extracting structured data from a text format. The difference is that HTML's structure is a tree of nested elements, not flat lines.

24.3 Beautiful Soup: Parsing HTML

Beautiful Soup is a Python library that takes an HTML document and turns it into a navigable tree of Python objects. Instead of writing regex to extract data from raw HTML text (which is fragile, error-prone, and painful), you use Beautiful Soup to search the tree by tag name, class, ID, or CSS selector.

24.3.1 Installation and First Parse

Beautiful Soup is a third-party library. Install it the way you learned in Chapter 23:

pip install beautifulsoup4 requests

The package name on PyPI is beautifulsoup4, but you import it as bs4:

from bs4 import BeautifulSoup

html = """
<html>
<head><title>Campus Events</title></head>
<body>
  <h1 class="page-title">Upcoming Events</h1>
  <div id="events">
    <div class="event" data-date="2025-09-15">
      <h2>Python Workshop</h2>
      <p class="description">Learn web scraping with Beautiful Soup.</p>
      <span class="location">Room 204, CS Building</span>
    </div>
    <div class="event" data-date="2025-09-18">
      <h2>Data Science Meetup</h2>
      <p class="description">Exploring pandas and matplotlib.</p>
      <span class="location">Auditorium B</span>
    </div>
  </div>
</body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

Two things to notice:

1. Always specify the parser. The second argument "html.parser" tells Beautiful Soup to use Python's built-in HTML parser. If you omit it, you'll get a GuessedAtParserWarning, and the behavior might differ across machines. Always be explicit.

2. soup is now a tree. You can navigate it, search it, and extract data from it using Beautiful Soup's methods.

24.3.2 Finding Elements: find() and find_all()

The two methods you'll use most:

# find() — returns the FIRST matching element (or None)
title = soup.find("title")
print(title.text)  # "Campus Events"

heading = soup.find("h1", class_="page-title")
print(heading.text)  # "Upcoming Events"

# find_all() — returns a LIST of ALL matching elements
events = soup.find_all("div", class_="event")
print(len(events))  # 2

You can search by tag name, by attribute, or by both:

# By tag name only
all_h2s = soup.find_all("h2")

# By class (note the underscore: class_ because "class" is a Python keyword)
descriptions = soup.find_all("p", class_="description")

# By ID (unique — so find() makes more sense than find_all())
events_container = soup.find("div", id="events")

# By custom attribute
dated_events = soup.find_all("div", attrs={"data-date": True})

Once you have a Tag object, you can extract its content:

for event in soup.find_all("div", class_="event"):
    name = event.find("h2").text              # Get visible text
    desc = event.find("p", class_="description").text
    location = event.find("span", class_="location").text
    date = event["data-date"]                 # Get an attribute value

    print(f"{name} ({date}) -- {location}")
    print(f"  {desc}")

Output:

Python Workshop (2025-09-15) -- Room 204, CS Building
  Learn web scraping with Beautiful Soup.
Data Science Meetup (2025-09-18) -- Auditorium B
  Exploring pandas and matplotlib.

Key extraction methods at a glance:

A catch with strip=True: it glues the stripped pieces together with nothing in between, so <p>Hello <b>world</b></p> becomes 'Helloworld'. Pass a separator as the first argument to keep words apart: tag.get_text(" ", strip=True) gives 'Hello world'.

Another surprise worth knowing about: class can hold several space-separated names, so Beautiful Soup always gives it back as a list. In our example, soup.find("h1")["class"] is ['page-title'], not 'page-title'. Most other attributes, such as href and data-date, come back as plain strings.

24.3.3 CSS Selectors: select() and select_one()

Beautiful Soup also supports CSS selectors — the same syntax used in web development to style elements. CSS selectors are often more concise for complex queries:

# By class
descriptions = soup.select(".description")

# By ID
events_div = soup.select_one("#events")

# Nested: spans inside elements with class "event"
locations = soup.select(".event .location")

# Tag with specific class
event_divs = soup.select("div.event")

# By attribute
dated = soup.select("div[data-date]")

# Attribute with specific value
sept15 = soup.select('div[data-date="2025-09-15"]')

Here's a quick reference for CSS selector syntax:

When to use find() vs. select()? Both work. find() and find_all() are Beautiful Soup's native API — slightly more verbose but readable, with keyword arguments for common attributes. select() uses CSS selector strings — more concise for complex queries, especially nested selections. Use whichever feels more natural for the task at hand.

📊 Decision Guide: find() vs. select()

Use find()/find_all() when...	Use select()/select_one() when...
You know the tag name and one attribute	You need nested selectors (".parent .child")
You want to pass keyword arguments	You're comfortable with CSS syntax
You're searching for a specific ID	You want concise multi-criteria queries
You want maximum readability for Python devs	You want maximum readability for web devs

🔄 Check Your Understanding 1: Given the HTML above, write Beautiful Soup code to extract just the location text from the second event. Do it two ways: once with find_all() and once with select(). Which approach do you find more readable?

24.4 Building a Web Scraper

Let's build a real scraper. We'll use http://quotes.toscrape.com — a website created specifically for practicing web scraping. It's designed for educational use, so we don't need to worry about overloading a production server.

24.4.1 Step 1: Fetch the Page

We need requests (from Chapter 21) to download the HTML, and Beautiful Soup to parse it:

import time
import requests
from bs4 import BeautifulSoup

def fetch_page(url: str, delay: float = 1.0) -> BeautifulSoup | None:
    """Fetch a web page and return a BeautifulSoup object.

    Args:
        url: The URL to fetch.
        delay: Seconds to wait before making the request (rate limiting).

    Returns:
        A BeautifulSoup object, or None if the request failed.
    """
    time.sleep(delay)  # Be polite -- wait before requesting

    headers = {
        "User-Agent": "CS1-Textbook-Example/1.0 (Educational purposes)"
    }

    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        return BeautifulSoup(response.text, "html.parser")
    except requests.exceptions.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

Three essentials every scraper should include:

1. time.sleep(delay) -- Rate limiting. Don't hammer the server (Section 24.5).
2. A descriptive User-Agent header -- The default python-requests/2.x is often blocked. Transparency is both polite and practical.
3. Error handling -- Networks fail, servers go down. try/except and timeout=10 ensure graceful failure.

24.4.2 Step 2: Parse and Extract

Now we need to understand the page's HTML structure. By inspecting quotes.toscrape.com in a browser (right-click, "Inspect"), we discover that each quote lives inside a <div class="quote">, with the text in <span class="text">, the author in <small class="author">, and tags inside <a class="tag"> elements.

def scrape_quotes(soup: BeautifulSoup) -> list[dict]:
    """Extract quotes, authors, and tags from a page."""
    quotes = []

    for quote_div in soup.select(".quote"):
        text = quote_div.select_one(".text").text
        text = text.strip("\u201c\u201d")  # Remove curly quote characters

        author = quote_div.select_one(".author").text

        tags = [tag.text for tag in quote_div.select(".tag")]

        quotes.append({
            "text": text,
            "author": author,
            "tags": tags,
        })

    return quotes

24.4.3 Step 3: Handle Pagination

Most websites split content across multiple pages. The quotes site has a "Next" button that links to the next page. We can follow it:

def get_next_page_url(soup: BeautifulSoup, base_url: str) -> str | None:
    """Find the URL of the next page, if it exists."""
    next_link = soup.select_one("li.next a")
    if next_link:
        return base_url + next_link["href"]
    return None

The "Next" link's href is a relative URL like /page/2/ — it says where to go on the same site, but not which site. Gluing it onto base_url gives http://quotes.toscrape.com/page/2/, which works here because every link on this site starts with /. On other sites, links can be written as page2.html or ../archive/, and simple concatenation produces broken URLs. The standard library's urllib.parse.urljoin(current_url, href) resolves every form correctly, just as a browser would:

from urllib.parse import urljoin

urljoin("http://quotes.toscrape.com/page/2/", "/page/3/")
# 'http://quotes.toscrape.com/page/3/'
urljoin("https://example.com/blog/post1.html", "post2.html")
# 'https://example.com/blog/post2.html'

Reach for it as soon as you scrape a site you didn't pick for practice.

24.4.4 Step 4: Put It All Together

def scrape_all_quotes(max_pages: int = 3) -> list[dict]:
    """Scrape quotes from multiple pages."""
    base_url = "http://quotes.toscrape.com"
    current_url = base_url
    all_quotes = []
    page_num = 1

    while current_url and page_num <= max_pages:
        print(f"Scraping page {page_num}: {current_url}")

        soup = fetch_page(current_url)
        if soup is None:
            print("  Failed to fetch page. Stopping.")
            break

        page_quotes = scrape_quotes(soup)
        all_quotes.extend(page_quotes)
        print(f"  Found {len(page_quotes)} quotes "
              f"(total: {len(all_quotes)})")

        current_url = get_next_page_url(soup, base_url)
        page_num += 1

    return all_quotes


# --- Run it ---
if __name__ == "__main__":
    quotes = scrape_all_quotes(max_pages=3)
    for q in quotes[:3]:
        print(f'"{q["text"]}" -- {q["author"]}')
        print(f"  Tags: {', '.join(q['tags'])}\n")

This scraper respects the server (1-second delays between requests), handles errors (try/except around network calls), stops after a configurable number of pages (no infinite crawling), and produces clean, structured data (list of dictionaries).

Notice the max_pages parameter. This is deliberate self-limitation. Even on a practice site, there's no reason to scrape every page when three pages demonstrate the concept. On real sites, this kind of restraint is both polite and practical.

✅ Best Practice: Defensive Scraping

Real websites change their HTML structure without warning. Your selectors that work today might break tomorrow. Build your scrapers defensively: - Check for None before accessing .text or attributes - Wrap extraction in try/except blocks - Log what you couldn't parse instead of crashing - Validate extracted data (are prices actually numbers? are dates valid?)

🔗 Connection -- Spaced Review (Ch 21): Notice how similar this is to the API pattern from Chapter 21 -- make a request, check for errors, extract data from the response. The difference is that APIs return structured JSON, while web pages return HTML that we have to parse. Same pattern, different data format.

🔄 Check Your Understanding 2: The scraper above uses max_pages=3 to limit how many pages it fetches. Why is this important? What could happen if you removed this limit and the site had 10,000 pages?

24.5 The Ethics of Scraping

Just because you can scrape a website doesn't mean you should. Web scraping sits in an ethical and legal gray area, and responsible practitioners follow a set of norms. This section isn't optional -- it's arguably the most important part of this chapter.

24.5.1 Check robots.txt First

The robots.txt file at the root of a website tells automated crawlers which parts of the site are off-limits. The format was an informal convention for almost thirty years before being written up as an internet standard, RFC 9309, in 2022. It's a voluntary standard -- nothing technically prevents you from ignoring it -- but ethical scrapers always respect it. Here's a typical one:

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/
Crawl-delay: 5

This says: all crawlers (User-agent: *) should avoid /admin/, /private/, and /api/internal/. The Crawl-delay: 5 asks you to wait at least 5 seconds between requests. Crawl-delay isn't part of RFC 9309, and some big crawlers (Google's, for one) ignore it, but it's the owner telling you their limit in plain numbers. Honor it.

You don't have to read these rules by eye. Python's standard library includes a parser for them:

from urllib.robotparser import RobotFileParser

robots = RobotFileParser("https://example.com/robots.txt")
robots.read()  # Downloads and parses the file

agent = "CS1-Textbook-Example/1.0"
print(robots.can_fetch(agent, "https://example.com/products/1"))  # True
print(robots.can_fetch(agent, "https://example.com/admin/users")) # False
print(robots.crawl_delay(agent))                                  # 5

(The output comments assume example.com serves the file shown above.)

What if a site has no robots.txt at all? Try http://quotes.toscrape.com/robots.txt in your browser: you'll get a 404 Not Found. Under RFC 9309, a missing file (a 4xx response) means the owner has set no crawling restrictions, and for a 404 can_fetch() returns True for everything. (Python's parser is stricter about two codes: if the server answers 401 Unauthorized or 403 Forbidden, it plays safe and can_fetch() returns False for everything.) That's not a license to hammer the site. Rate limits and the Terms of Service still apply.

Always respect robots.txt. If a path is disallowed, don't scrape it. Period.

⚠️ Important: robots.txt is a voluntary standard, not a technical barrier. Your scraper can ignore it -- but it shouldn't. Ignoring robots.txt is like ignoring a "Do Not Enter" sign on an unlocked door. You can walk in, but you're violating the owner's stated wishes.

24.5.2 Rate Limiting: Don't Be a Bully

A web server is a shared resource. When you send 100 requests per second, you're consuming server capacity that real users need. Aggressive scraping can slow down a website -- or crash it entirely for a small site.

Rule of thumb: add at least time.sleep(1) between requests. If robots.txt specifies a Crawl-delay, use that instead. When in doubt, scrape slower.

import time

for url in urls_to_scrape:
    time.sleep(2)  # Wait 2 seconds between requests
    response = requests.get(url, headers=headers, timeout=10)
    # process response...

Guidelines for responsible rate limiting: - Minimum 1 second between requests to the same server - Respect Crawl-delay in robots.txt if specified - Back off if you receive 429 (Too Many Requests) or 503 (Service Unavailable) status codes - Consider the site's size -- a personal blog can handle far less traffic than Amazon

24.5.3 Terms of Service and Legal Considerations

Many websites explicitly address scraping in their Terms of Service. Read the ToS. If they prohibit scraping, your options are: don't scrape, use their API instead, or contact the site owner for permission.

The legal landscape is evolving, and the best-known US case shows why "public" doesn't mean "anything goes." In hiQ Labs v. LinkedIn, the Ninth Circuit Court of Appeals ruled (in 2019, and again in 2022) that scraping publicly viewable profiles likely doesn't count as "unauthorized access" under the Computer Fraud and Abuse Act. Yet hiQ still lost: in late 2022 a federal court found that its scraping breached LinkedIn's User Agreement, and the case settled with hiQ agreeing to stop scraping and delete the data. Privacy law adds another layer. The EU's GDPR applies to personal data even when it's publicly visible. California's CCPA exempts some publicly available information, but not everything that happens to be online. Scraping behind login walls or circumventing access controls is much riskier still.

The safe path: scrape public data, respect robots.txt and ToS, rate-limit aggressively, and use APIs when available.

24.5.4 The Ethical Scraping Checklist

Before you scrape any website, work through this checklist:

⚖️ Ethical Analysis: "It's technically public" is not the same as "it's ethical to bulk-collect." Aggregating personal data -- even from public profiles -- can cause real harm. Consider the reasonable expectations of the people whose data you're collecting, not just what's technically accessible. See Case Study 2 for five scenarios that explore this boundary in depth.

🔄 Check Your Understanding 3: A website's robots.txt allows scraping of /products/ but its Terms of Service say "no automated access." What should you do? Does robots.txt permission override the Terms of Service?

24.6 Automation Beyond Scraping

Web scraping is one form of automation, but Python can automate virtually any repetitive computer task. Let's look at some of the most common patterns.

24.6.1 File Organization

Your Downloads folder is a mess. Here's a script that sorts files into subdirectories by type:

import shutil
from pathlib import Path

def organize_downloads(download_dir: Path) -> dict[str, int]:
    """Sort files into subdirectories by extension."""
    extension_map = {
        ".pdf": "documents", ".docx": "documents", ".txt": "documents",
        ".jpg": "images", ".png": "images", ".gif": "images",
        ".mp3": "audio", ".wav": "audio",
        ".py": "code", ".js": "code", ".html": "code",
        ".csv": "data", ".json": "data", ".xlsx": "data",
    }

    moved_counts: dict[str, int] = {}

    # list() takes a snapshot first, so we aren't reading the folder
    # while we move files out of it and create new subfolders
    for file_path in list(download_dir.iterdir()):
        if file_path.is_file():
            ext = file_path.suffix.lower()
            folder_name = extension_map.get(ext, "other")

            dest_dir = download_dir / folder_name
            dest_dir.mkdir(exist_ok=True)

            shutil.move(str(file_path), str(dest_dir / file_path.name))
            moved_counts[folder_name] = moved_counts.get(folder_name, 0) + 1

    return moved_counts

One gap to know about: if documents/ already contains a notes.txt, moving a second notes.txt there silently overwrites the old one. The version in code/file_automation.py checks for this and adds a number to the new name instead.

24.6.2 Batch File Renaming

You took 200 photos named IMG_4382.jpg, IMG_4383.jpg, etc. You want rome_001.jpg, rome_002.jpg:

from pathlib import Path

def batch_rename(directory: Path, prefix: str, dry_run: bool = True):
    """Rename files with a consistent numbering scheme."""
    files = sorted(f for f in directory.iterdir() if f.is_file())

    for i, file_path in enumerate(files, start=1):
        new_name = f"{prefix}_{i:03d}{file_path.suffix}"
        new_path = file_path.parent / new_name

        if new_path.exists() and new_path != file_path:
            print(f"  SKIPPED: {new_name} already exists")
        elif dry_run:
            print(f"  [DRY RUN] {file_path.name} -> {new_name}")
        else:
            file_path.rename(new_path)
            print(f"  Renamed: {file_path.name} -> {new_name}")

Notice the dry_run parameter -- a professional practice that lets you preview what will happen before committing to irreversible file operations.

The exists() check matters more than it looks. Say you rename the Rome photos, then copy in one more, IMG_5000.jpg. Sorted by name, it comes first (I sorts before r), so the script wants to call it rome_001.jpg, a name that's already taken. On Windows, rename() refuses and raises FileExistsError. On macOS and Linux, it silently replaces the old rome_001.jpg, and that photo is gone. Never let a file operation overwrite something without checking first. The check prevents data loss, but a second run over a half-renamed folder can still shuffle the numbers around, which is exactly the kind of surprise a dry run shows you before anything happens.

24.6.3 Automated Report Generation

Reports from any data source follow the same pattern: load data, compute statistics, format output, save to file.

from datetime import datetime
from pathlib import Path

def generate_directory_report(target_dir: Path) -> str:
    """Generate a summary report of a directory's contents."""
    file_count = 0
    total_size = 0
    extension_counts: dict[str, int] = {}

    for item in target_dir.rglob("*"):
        if item.is_file():
            file_count += 1
            total_size += item.stat().st_size
            ext = item.suffix.lower() or "(no extension)"
            extension_counts[ext] = extension_counts.get(ext, 0) + 1

    lines = [
        f"Directory Report: {target_dir}",
        f"Generated: {datetime.now():%Y-%m-%d %H:%M:%S}",
        "=" * 50,
        f"Total files: {file_count:,}",
        f"Total size: {total_size / 1024:.1f} KB",
        "",
        "By file type:",
    ]
    for ext, count in sorted(
        extension_counts.items(), key=lambda x: x[1], reverse=True
    ):
        lines.append(f"  {ext:15s} {count:>5} files")

    return "\n".join(lines)

🔗 Connection -- Spaced Review (Ch 10): The file I/O patterns here -- Path.mkdir(), Path.iterdir(), Path.rglob(), Path.stat() -- are the same pathlib tools you learned in Chapter 10. Automation is rarely about learning new tools; it's about combining familiar tools into workflows that save real time.

24.7 Scheduling Scripts

Writing an automation script is great. Having it run automatically -- without you remembering to execute it -- is even better. There are three main approaches.

24.7.1 cron (Linux/macOS)

cron is the standard Unix job scheduler. You define when to run a command using a five-field syntax:

# +------------ minute (0-59)
# | +---------- hour (0-23)
# | | +-------- day of month (1-31)
# | | | +------ month (1-12)
# | | | | +---- day of week (0-7, 0 and 7 are Sunday)
# | | | | |
# * * * * *  command

To edit your cron schedule:

crontab -e

Common examples:

# Every day at 8:00 AM
0 8 * * * /usr/bin/python3 /home/elena/reports/daily_report.py

# Every Friday at 6:00 AM
0 6 * * 5 /usr/bin/python3 /home/elena/reports/weekly_pipeline.py

# Every 30 minutes
*/30 * * * * /usr/bin/python3 /home/elena/scripts/check_updates.py

# First day of every month at midnight
0 0 1 * * /usr/bin/python3 /home/elena/scripts/monthly_cleanup.py

💡 Tip: The website crontab.guru lets you type a cron expression and see a plain-English explanation of when it will run. Bookmark it.

Two important details. First, use full paths for both Python and your script. cron runs your command in a bare-bones environment: it doesn't know your PATH, and it never activates your virtual environment. If your script needs packages you installed in a venv (Chapter 23), point cron at the venv's own Python, which always uses that venv's packages. Second, redirect output to a log file so you can see what went wrong when a run fails at 6 AM with nobody watching:

0 6 * * 5 /home/elena/reports/.venv/bin/python /home/elena/reports/weekly_pipeline.py >> /home/elena/reports/pipeline.log 2>&1

>> appends the script's normal output to the log, and 2>&1 sends error messages (tracebacks included) to the same place.

24.7.2 Task Scheduler (Windows)

Windows has a GUI-based Task Scheduler and a command-line tool called schtasks:

schtasks /create /tn "Weekly Report" /tr "C:\Users\Elena\reports\.venv\Scripts\python.exe C:\Users\Elena\reports\weekly_pipeline.py" /sc weekly /d FRI /st 06:00

This creates a task named "Weekly Report" (/tn) that runs the given command (/tr) weekly (/sc weekly) on Fridays (/d FRI) at 6:00 AM (/st 06:00). The full-path advice from cron applies here too: name the venv's python.exe directly.

24.7.3 The schedule Library (Cross-Platform)

If you want scheduling that's simpler than cron and works on any operating system, the schedule library provides a clean Python API:

pip install schedule

import schedule
import time

def daily_report():
    print("Generating daily report...")
    # your report logic here

def check_for_updates():
    print("Checking for updates...")
    # your update logic here

schedule.every().day.at("08:00").do(daily_report)
schedule.every(30).minutes.do(check_for_updates)
schedule.every().friday.at("06:00").do(daily_report)

# This loop must keep running for the schedule to work
while True:
    schedule.run_pending()
    time.sleep(60)  # Check every minute

The trade-off: schedule is simpler to set up, but it requires your Python process to stay running. If the process crashes or the computer restarts, the schedule stops. For production automation, cron or Task Scheduler is more robust because they're managed by the operating system.

🔄 Check Your Understanding 4: Elena wants her report to run every Friday at 6 AM on her macOS laptop. Write the cron expression. What happens if her laptop is turned off at 6 AM on Friday?

24.8 A Complete Automation Pipeline

Let's put everything together. The general pattern for any automation project is Fetch, Parse, Transform, Output -- four stages, each with a clear responsibility.

+-----------+    +-----------+    +-----------+    +-----------+
|   FETCH   | -> |   PARSE   | -> | TRANSFORM | -> |  OUTPUT   |
|           |    |           |    |           |    |           |
| Download  |    | Extract   |    | Clean,    |    | Save to   |
| HTML, CSV |    | data from |    | compute,  |    | CSV/JSON, |
| or API    |    | raw HTML  |    | validate  |    | report    |
+-----------+    +-----------+    +-----------+    +-----------+

Here's a simplified version of Elena's pipeline. Each function has one job, and each function's output feeds the next. To keep it runnable without a network connection, the "fetch" step starts from CSV text already in memory (SAMPLE_DATA, defined in the full version), so fetch_data() only has to parse it:

"""Elena's automated report pipeline -- simplified."""

import csv
import json
import time
from datetime import datetime
from pathlib import Path


def fetch_data(csv_text: str) -> list[dict]:
    """Fetch + Parse: turn raw CSV text into a list of records."""
    reader = csv.DictReader(csv_text.strip().splitlines())
    return [
        {"county": row["county"], "service": row["service"],
         "clients": int(row["clients_served"]),
         "hours": int(row["hours_delivered"])}
        for row in reader
    ]


def validate(records: list[dict]) -> tuple[list[dict], list[str]]:
    """Transform, part 1: separate clean records from bad ones."""
    clean, errors = [], []
    for r in records:
        if r["clients"] < 0:
            errors.append(f"Negative clients: {r}")
        elif r["hours"] < 0:
            errors.append(f"Negative hours: {r}")
        else:
            clean.append(r)
    return clean, errors


def summarize(records: list[dict]) -> dict:
    """Transform, part 2: compute summary statistics."""
    by_county: dict[str, dict] = {}
    for r in records:
        county = r["county"]
        if county not in by_county:
            by_county[county] = {"clients": 0, "hours": 0}
        by_county[county]["clients"] += r["clients"]
        by_county[county]["hours"] += r["hours"]

    return {
        "generated": datetime.now().isoformat(),
        "total_clients": sum(r["clients"] for r in records),
        "total_hours": sum(r["hours"] for r in records),
        "by_county": by_county,
    }


def save_report(summary: dict, output_dir: Path) -> Path:
    """Output: format the report and save it to a file."""
    output_dir.mkdir(parents=True, exist_ok=True)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")

    lines = [
        "=" * 55,
        "  HARBOR COMMUNITY SERVICES -- IMPACT REPORT",
        f"  Generated: {summary['generated'][:19]}",
        f"  Clients: {summary['total_clients']:,}",
        f"  Hours:   {summary['total_hours']:,}",
        "",
    ]
    for county, data in sorted(summary["by_county"].items()):
        lines.append(f"  {county:15s} {data['clients']:>5,} clients"
                     f"  {data['hours']:>5,} hours")
    lines.append("=" * 55)

    path = output_dir / f"report_{timestamp}.txt"
    path.write_text("\n".join(lines), encoding="utf-8")
    return path


def run_pipeline():
    """Execute the full pipeline."""
    start = time.time()
    records = fetch_data(SAMPLE_DATA)
    clean, errors = validate(records)
    for error in errors:             # Never drop bad rows silently
        print(f"WARNING: {error}")
    summary = summarize(clean)
    path = save_report(summary, Path("reports"))
    elapsed = time.time() - start
    print(f"Report saved: {path} ({elapsed:.2f}s)")
    print("Elena's old process: ~4 hours.")

The full version with sample data is in code/automation_pipeline.py; Elena's complete five-stage pipeline is in Case Study 1.

Why this structure matters. Each stage has one job. If the data format changes, you modify fetch_data(). If validation rules change, you modify validate(). If the report format changes, you modify save_report(). The stages are independent, testable, and replaceable -- the same design principles you learned in Chapter 6 (functions) and Chapter 12 (modules).

This architecture gives you testability (test each stage independently with known data), debuggability (you know which stage failed), and extensibility (add a new data source to the fetch stage or a new output format to the output stage without changing the others).

🔗 Connection -- Spaced Review (Ch 12): This pipeline is the module pattern from Chapter 12 applied to data processing. Each stage could live in its own module: fetch.py, validate.py, transform.py, output.py.

🔗 Connection -- Spaced Review (Ch 18): Scraping a website with nested categories is naturally recursive -- follow links to sub-pages, which have their own links, until a base case (maximum depth or no more links). This is tree traversal from Chapter 18, with one twist: a website isn't a perfect tree, because pages link back to each other. Keep a set of URLs you've already visited, or your crawler will go around in circles forever.

24.9 When NOT to Scrape

Knowing when not to scrape is as important as knowing how. Here's when you should reach for something else.

24.9.1 Use an API Instead

If a website offers an API, use it. APIs return clean JSON with documented fields and versioned endpoints. Scraped HTML depends on the page's visual structure, which can change any time the site redesigns. Scraping when an API exists is like climbing through a window when the front door is open.

24.9.2 Other Situations to Avoid

If a page's content is rendered by JavaScript, Beautiful Soup won't see it -- it only gets the raw HTML before JavaScript runs. Before reaching for a headless browser like Selenium, check the browser's Network tab for XHR/fetch requests. Often the data is loaded from a JSON endpoint you can query directly.

💡 Intuition: If you have to work hard to get the data -- evading bot detection, rotating proxies, solving CAPTCHAs -- that's the website telling you "no."

24.10 Project Checkpoint: TaskFlow v2.3

It's time to add automation features to TaskFlow. Building on v2.2 (Chapter 23, which added a virtual environment and rich terminal output), we'll add two new capabilities:

1. Daily motivational quote scraper -- Fetch a random quote from quotes.toscrape.com and display it when the app starts, with caching so we don't re-scrape unnecessarily.
2. Automated task report generation -- Generate a formatted status report of all tasks and save it to a timestamped file.

24.10.1 Feature 1: Daily Quote Scraper

The quote scraper demonstrates every pattern from this chapter in miniature: fetch a page, parse HTML, extract data, handle errors, and cache the result to avoid unnecessary requests.

You may remember that v2.0 (Chapter 21) already showed a quote at startup, fetched as tidy JSON from an API. By Section 24.9's rules, the API version is the better design, and in a real project you'd keep it. We scrape here because a practice site is a safe place to practice, and pulling the same kind of data out of raw HTML shows you exactly how much extra work, and fragility, the HTML route adds.

import json
from datetime import datetime
from pathlib import Path

# Third-party (pip install requests beautifulsoup4)
try:
    import requests
    from bs4 import BeautifulSoup
    HAS_WEB_LIBS = True
except ImportError:
    HAS_WEB_LIBS = False

# Every file lives next to the script, like taskflow_data.json
HERE = Path(__file__).parent
QUOTE_CACHE = HERE / "taskflow_quote_cache.json"


def fetch_daily_quote() -> dict | None:
    """Scrape a random quote, with daily caching."""
    if not HAS_WEB_LIBS:
        return None

    # Check cache -- don't re-scrape if we have today's quote
    if QUOTE_CACHE.exists():
        try:
            cache = json.loads(QUOTE_CACHE.read_text(encoding="utf-8"))
            if cache.get("date") == datetime.now().strftime("%Y-%m-%d"):
                return cache.get("quote")
        except (json.JSONDecodeError, KeyError):
            pass

    # Fetch a new quote
    try:
        headers = {"User-Agent": "TaskFlow/2.3 (Educational project)"}
        response = requests.get(
            "http://quotes.toscrape.com/random",
            headers=headers, timeout=10,
        )
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        quote_div = soup.select_one(".quote")

        if quote_div:
            text = quote_div.select_one(".text").text.strip("\u201c\u201d")
            author = quote_div.select_one(".author").text
            quote = {"text": text, "author": author}

            # Cache for today
            QUOTE_CACHE.write_text(json.dumps(
                {"date": datetime.now().strftime("%Y-%m-%d"), "quote": quote},
                indent=2), encoding="utf-8")
            return quote

    except requests.exceptions.RequestException as e:
        print(f"  Could not fetch quote: {e}")

    return None  # Caller displays a fallback offline quote

The caching pattern is worth noting: check the cache before making a network request, and if the network is down, degrade gracefully rather than crashing. The app works with or without internet.

24.10.2 Feature 2: Automated Task Report

The report generator follows the transform-output pattern from Section 24.8 -- compute summary statistics from the task list, format a text report, and save it with a timestamped filename:

def generate_task_report(tasks: list[dict]) -> str:
    """Generate a formatted task status report."""
    done = sum(1 for t in tasks if t.get("completed"))
    lines = [
        "=" * 55,
        "  TASKFLOW -- TASK STATUS REPORT",
        f"  Generated: {datetime.now():%Y-%m-%d %H:%M:%S}",
        f"  Total tasks: {len(tasks)} ({done} completed, {len(tasks) - done} pending)",
        "=" * 55, "",
    ]

    for priority in ["high", "medium", "low"]:
        count = sum(1 for t in tasks if t.get("priority") == priority)
        lines.append(f"  {priority:8s}  {count:>3}  {'#' * count}")
    lines.append("")

    for i, task in enumerate(tasks, 1):
        status = "[x]" if task.get("completed") else "[ ]"
        lines.append(f"  {i:>3}. {status} {task.get('title', 'Untitled')}"
                     f" ({task.get('priority', 'medium')}, "
                     f"{task.get('category', 'general')})")

    return "\n".join(lines)

The tasks are the same dictionaries every version since v2.0 has saved, with the keys v1.5's Task.to_dict() writes, so the report reads "title" and the "completed" flag. Like v2.0-v2.2, the checkpoint is a stand-alone script, and its task commands are trimmed to add, list, and complete so the scraping and report code stands out. The quote cache, the data file, and the taskflow_reports/ folder all live next to the script.

The menu integration is straightforward: display a daily quote on startup (with an offline fallback), and add a "Generate report" option that calls generate_task_report(), prints the result, and saves it. The complete code is in code/project-checkpoint.py.

24.10.3 What Changed from v2.2 to v2.3

🔗 Spaced Review (Ch 12): TaskFlow's modular structure from v1.1 pays off again. The quote scraper and report generator are self-contained functions that slot into the existing codebase without modifying the core task management logic. Good module boundaries make features additive, not invasive.

🔗 Spaced Review (Ch 22): In v2.1, you added regex-powered search to TaskFlow. If you wanted to validate scraped data -- say, checking that a quote's author name contains only expected characters -- regex would be the natural tool. Be careful what you call "expected," though: re.fullmatch(r"[A-Za-z .'-]+", author) would reject André Gide, who appears on the site's very first page. Python's \w matches letters from every alphabet, so re.fullmatch(r"[\w .'-]+", author) is the safer check to combine with the Beautiful Soup extraction.

24.11 Debugging Walkthrough

🐛 Debugging Walkthrough: Common Scraping Errors

Symptom: GuessedAtParserWarning: No parser was explicitly specified Cause: You wrote BeautifulSoup(html) without specifying a parser. Fix: Always pass the parser: BeautifulSoup(html, "html.parser"). This makes your code portable across different machines.

Symptom: Your scraper returns empty results, even though you can see data in the browser. Cause 1: The page uses JavaScript to load content dynamically. requests.get() fetches the initial HTML, but JavaScript-rendered content isn't in the raw source. Fix: Check "View Page Source" (not "Inspect Element" -- they show different things). If the data isn't in the source, you need Selenium or Playwright, or check if the site has an API. Look in the browser's Network tab for XHR/fetch requests -- the data might be loaded from a JSON endpoint you can query directly. Try it: http://quotes.toscrape.com/js/ shows the same quotes as the home page in a browser, but soup.select(".quote") on what requests downloads returns an empty list. View its source and you'll find the quotes sitting inside a <script> tag as JavaScript data, waiting for the browser to build the page. Cause 2: Your CSS selector doesn't match. Maybe the site uses class="product-card" and you're selecting ".product_card" (underscore instead of hyphen). Fix: Copy the exact class name from the browser's Inspect panel. Test your selector on a small HTML snippet first.

Symptom: AttributeError: 'NoneType' object has no attribute 'text' Cause: find() or select_one() returned None because no element matched, and you tried to call .text on it. Fix: Always check for None before accessing attributes: python element = soup.select_one(".price") if element: price = element.text else: price = "N/A" Or use the walrus operator for concise conditional extraction: python price = el.text if (el := soup.select_one(".price")) else "N/A"

Symptom: ConnectionError or Timeout when fetching pages. Cause: Network issues, or the server is blocking your requests (no User-Agent, too many requests too fast). Fix: Add a descriptive User-Agent header, increase time.sleep() between requests, add timeout=10 to requests.get(), and wrap everything in try/except.

Symptom: The scraper works for a few days, then suddenly breaks. Cause: The website redesigned its HTML structure. CSS class names changed or elements were rearranged. Fix: This is inherent to scraping. Document your assumptions, add validation to detect structure changes, and consider using an API if available. Scrapers require ongoing maintenance.

Chapter Summary

This chapter covered two of the most practical skills in programming: web scraping and automation.

You learned to parse HTML with Beautiful Soup -- searching by tag, class, ID, or CSS selector to extract structured data from web pages. You built a complete web scraper with error handling, rate limiting, and pagination. You confronted the ethics of scraping -- robots.txt, Terms of Service, and the question of when scraping is responsible. You explored automation beyond scraping -- file organization, batch renaming, and report generation. You learned to schedule scripts with cron, Task Scheduler, and the schedule library. And you saw the pipeline pattern: Fetch, Parse, Transform, Output.

The threshold concept -- automation as a multiplier -- is this: every hour spent writing an automation script pays for itself many times over. Elena spent 8 hours building her pipeline. It saves her 200 hours per year. But more than speed, the script is reliable. It doesn't get tired, doesn't make transcription errors, and doesn't forget to validate the data. The best automation isn't just faster -- it's better.

Spaced Review

Concepts from earlier chapters that appeared in this chapter:

What's Next

In Chapter 25, you'll learn version control with Git -- how to track your code's history, experiment fearlessly on branches, and collaborate with other developers. You'll initialize a Git repository for TaskFlow and learn the workflow that powers open-source development worldwide.

"The first rule of automation: if you have to do it twice, write a script. The second rule: if the script works, schedule it."