Exercises: Web Scraping and Automation

These exercises progress from HTML parsing fundamentals through complete automation pipelines. For scraping exercises, use the practice site http://quotes.toscrape.com or work with local HTML strings. Never scrape a site without checking its robots.txt first.

Difficulty Guide: - ⭐ Foundational (5-10 min each) - ⭐⭐ Intermediate (10-20 min each) - ⭐⭐⭐ Challenging (20-40 min each) - ⭐⭐⭐⭐ Advanced/Research (40+ min each)

Part A: HTML Parsing Basics ⭐

A.1. Given the following HTML string, use Beautiful Soup to extract and print the text of every <li> element:

html = """
<ul id="languages">
  <li class="compiled">C++</li>
  <li class="interpreted">Python</li>
  <li class="compiled">Rust</li>
  <li class="interpreted">JavaScript</li>
  <li class="interpreted">Ruby</li>
</ul>
"""

Expected output:

C++
Python
Rust
JavaScript
Ruby

A.2. Using the same HTML from A.1, extract only the interpreted languages (those with class="interpreted"). Use find_all() with the class_ parameter.

A.3. Parse the following HTML and extract the href attribute from every <a> tag:

html = """
<nav>
  <a href="/home">Home</a>
  <a href="/about">About</a>
  <a href="/contact">Contact</a>
  <a href="https://example.com/external">External Link</a>
</nav>
"""

Print each link's text and URL. Which links are relative and which are absolute?

A.4. Write a function extract_table_data(html: str) -> list[list[str]] that parses an HTML table and returns its contents as a list of lists. Test it with:

html = """
<table>
  <tr><th>Name</th><th>Score</th><th>Grade</th></tr>
  <tr><td>Alice</td><td>95</td><td>A</td></tr>
  <tr><td>Bob</td><td>82</td><td>B</td></tr>
  <tr><td>Carol</td><td>78</td><td>C+</td></tr>
</table>
"""

Expected output:

[['Name', 'Score', 'Grade'],
 ['Alice', '95', 'A'],
 ['Bob', '82', 'B'],
 ['Carol', '78', 'C+']]

Part B: CSS Selectors ⭐-⭐⭐

B.1. Rewrite exercises A.1 and A.2 using soup.select() and CSS selector syntax instead of find_all(). Which approach do you find more readable?

B.2. Given this HTML, use CSS selectors to extract only the prices of items that are in stock:

html = """
<div class="catalog">
  <div class="product in-stock">
    <h3>Widget A</h3>
    <span class="price">$12.99</span>
  </div>
  <div class="product out-of-stock">
    <h3>Widget B</h3>
    <span class="price">$8.49</span>
  </div>
  <div class="product in-stock">
    <h3>Widget C</h3>
    <span class="price">$24.99</span>
  </div>
  <div class="product in-stock">
    <h3>Widget D</h3>
    <span class="price">$5.99</span>
  </div>
</div>
"""

Hint: The CSS selector .in-stock .price selects .price elements inside .in-stock elements.

B.3. Write a function count_elements(html: str, selector: str) -> int that takes an HTML string and a CSS selector, and returns the number of matching elements. Test with several selectors on the product catalog HTML from B.2.

B.4. Parse the following nested HTML and extract each student's name and all of their course names:

html = """
<div class="roster">
  <div class="student" data-id="1">
    <h2>Priya Sharma</h2>
    <ul class="courses">
      <li>CS 101</li>
      <li>MATH 201</li>
      <li>PHYS 101</li>
    </ul>
  </div>
  <div class="student" data-id="2">
    <h2>Diego Alvarez</h2>
    <ul class="courses">
      <li>CS 101</li>
      <li>ENG 102</li>
    </ul>
  </div>
</div>
"""

Output should be a dictionary: {"Priya Sharma": ["CS 101", "MATH 201", "PHYS 101"], "Diego Alvarez": ["CS 101", "ENG 102"]}.

Part C: Web Scraping ⭐⭐-⭐⭐⭐

C.1. Write a scraper that fetches the first page of http://quotes.toscrape.com and prints each quote along with its author. Include a 1-second delay before the request and a descriptive User-Agent header.

C.2. Extend C.1 to also extract the tags for each quote. Group the output by tag: for each tag, list all quotes that have that tag.

C.3. Write a scraper that follows the "Next" links through http://quotes.toscrape.com/tag/life/ (quotes tagged "life"), stopping after 3 pages or when there is no "Next" link, whichever comes first, and collects all unique authors. Print the authors sorted alphabetically. How many pages did your scraper actually fetch? (Watch the "Next" link's href on this tag page: does base_url + href still give the right URL?)

C.4. Write a function check_robots_txt(url: str) -> str that fetches the robots.txt file for a given domain and returns its text, or a clear message if the site doesn't have one. Test it with http://quotes.toscrape.com/robots.txt and with one other site of your choice. Check response.status_code. What does a 404 response tell you about which paths a crawler may visit (Section 24.5.1)? Then use urllib.robotparser.RobotFileParser to ask whether your scraper may fetch /page/2/ on each site.

Part D: File Automation ⭐⭐

D.1. Write a script that scans a directory and generates a report showing: - Total number of files - Total size (in human-readable format: KB, MB, etc.) - Breakdown by file extension - The 5 most recently modified files

Test it on your own project directory.

D.2. Write a function find_duplicates(directory: Path) -> list[list[Path]] that finds files with identical sizes in a directory. (For a simple heuristic, files with the same size might be duplicates. A production version would compare file hashes, but size comparison is fine for this exercise.)

D.3. Write a script that monitors a directory for new files. Every 5 seconds, it checks whether any new files have appeared and prints a notification. Use time.sleep(5) in a loop. (Hint: compare the current set of files to the set from the previous check.)

Part E: Automation Pipelines ⭐⭐⭐

E.1. Build a mini pipeline that: 1. Reads a CSV file of student grades 2. Validates that all grades are between 0 and 100 3. Computes the mean, median, highest, and lowest score for each assessment (exam1, exam2, final); the standard library's statistics.mean() and statistics.median() will help 4. Generates a formatted text report 5. Saves the report to a file with a timestamped filename

Use this sample CSV data:

name,exam1,exam2,final
Alice,92,88,95
Bob,78,82,71
Carol,95,91,98
David,65,70,68
Eve,88,85,90

E.2. Create an automation script that generates a "daily digest" file every time it runs. The digest should include: - Current date and time - A quote (scraped or from a local list) - A summary of your TaskFlow tasks (read from JSON) - Weather data placeholder (print "Weather: [integrate API from Ch 21]")

Save each digest to a digests/ directory with the filename digest_YYYYMMDD.txt.

E.3. (Spaced review — Chapter 21, CSV and JSON) Modify the scraper from C.1 to save its results as both JSON and CSV files. Compare: which format is better for this data, and why? (Think about nested data like tags.)

Part F: Integration and Design ⭐⭐⭐⭐

F.1. Build a "link checker" that takes an HTML file (or string) and checks whether all the href links in <a> tags return a 200 status code. Turn relative links into full URLs with urljoin() first (Section 24.4.3), and skip mailto: links. For each link, report whether it's valid, broken (4xx), a server error (5xx), or unreachable (connection error). Add rate limiting with time.sleep(1) between requests.

F.2. (Spaced review — Chapter 18, Recursion) Write a recursive function scrape_site_map(url: str, max_depth: int = 2) -> dict that, given a starting URL, finds all internal links on the page, then follows those links to find more internal links, up to max_depth levels deep. Return a nested dictionary representing the site structure. Use http://quotes.toscrape.com as your test site. Be sure to include rate limiting and avoid visiting the same URL twice. Also cap the total number of pages you fetch (50 is plenty while you're testing): the home page alone links to 46 other pages on the site, and two levels deep reaches roughly 150 different pages. Every bug you fix would cost you another couple of minutes of polite one-second waits.

F.3. Design (but don't necessarily implement) an automation system that keeps Dr. Patel's local collection of protein and sequence records up to date every week. Write pseudocode and a design document that covers: - What data to collect and from where. Big public bioinformatics databases, such as UniProt (which she queried in Chapter 21), offer APIs. Justify whether any part of the job needs scraping at all. - Rate limiting strategy - Error handling and retry logic - Data storage format - Scheduling approach - Ethical considerations specific to scientific data

This is a design exercise — focus on the thinking, not the code.