Exercises: Web Scraping and Automation

Contributors

Exercises: Web Scraping and Automation

These exercises progress from HTML parsing fundamentals through complete automation pipelines. For scraping exercises, use the practice site http://quotes.toscrape.com or work with local HTML strings. Never scrape a site without checking its robots.txt first.

Difficulty Guide: - ⭐ Foundational (5-10 min each) - ⭐⭐ Intermediate (10-20 min each) - ⭐⭐⭐ Challenging (20-40 min each) - ⭐⭐⭐⭐ Advanced/Research (40+ min each)

Part A: HTML Parsing Basics ⭐

A.1. Given the following HTML string, use Beautiful Soup to extract and print the text of every <li> element:

html = """
<ul id="languages">
  <li class="compiled">C++</li>
  <li class="interpreted">Python</li>
  <li class="compiled">Rust</li>
  <li class="interpreted">JavaScript</li>
  <li class="interpreted">Ruby</li>
</ul>
"""

Expected output:

C++
Python
Rust
JavaScript
Ruby

A.2. Using the same HTML from A.1, extract only the interpreted languages (those with class="interpreted"). Use find_all() with the class_ parameter.

A.3. Parse the following HTML and extract the href attribute from every <a> tag:

html = """
<nav>
  <a href="/home">Home</a>
  <a href="/about">About</a>
  <a href="/contact">Contact</a>
  <a href="https://example.com/external">External Link</a>
</nav>
"""

Print each link's text and URL. Which links are relative and which are absolute?

A.4. Write a function extract_table_data(html: str) -> list[list[str]] that parses an HTML table and returns its contents as a list of lists. Test it with:

html = """
<table>
  <tr><th>Name</th><th>Score</th><th>Grade</th></tr>
  <tr><td>Alice</td><td>95</td><td>A</td></tr>
  <tr><td>Bob</td><td>82</td><td>B</td></tr>
  <tr><td>Carol</td><td>78</td><td>C+</td></tr>
</table>
"""

Expected output:

[['Name', 'Score', 'Grade'],
 ['Alice', '95', 'A'],
 ['Bob', '82', 'B'],
 ['Carol', '78', 'C+']]

Part B: CSS Selectors ⭐-⭐⭐

B.1. Rewrite exercises A.1 and A.2 using soup.select() and CSS selector syntax instead of find_all(). Which approach do you find more readable?

B.2. Given this HTML, use CSS selectors to extract only the prices of items that are in stock:

html = """
<div class="catalog">
  <div class="product in-stock">
    <h3>Widget A</h3>
    <span class="price">$12.99</span>
  </div>
  <div class="product out-of-stock">
    <h3>Widget B</h3>
    <span class="price">$8.49</span>
  </div>
  <div class="product in-stock">
    <h3>Widget C</h3>
    <span class="price">$24.99</span>
  </div>
  <div class="product in-stock">
    <h3>Widget D</h3>
    <span class="price">$5.99</span>
  </div>
</div>
"""

Hint: The CSS selector .in-stock .price selects .price elements inside .in-stock elements.

B.3. Write a function count_elements(html: str, selector: str) -> int that takes an HTML string and a CSS selector, and returns the number of matching elements. Test with several selectors on the product catalog HTML from B.2.

B.4. Parse the following nested HTML and extract each student's name and all of their course names:

html = """
<div class="roster">
  <div class="student" data-id="1">
    <h2>Priya Sharma</h2>
    <ul class="courses">
      <li>CS 101</li>
      <li>MATH 201</li>
      <li>PHYS 101</li>
    </ul>
  </div>
  <div class="student" data-id="2">
    <h2>Marcus Chen</h2>
    <ul class="courses">
      <li>CS 101</li>
      <li>ENG 102</li>
    </ul>
  </div>
</div>
"""

Output should be a dictionary: {"Priya Sharma": ["CS 101", "MATH 201", "PHYS 101"], "Marcus Chen": ["CS 101", "ENG 102"]}.

Part C: Web Scraping ⭐⭐-⭐⭐⭐

C.1. Write a scraper that fetches the first page of http://quotes.toscrape.com and prints each quote along with its author. Include a 1-second delay before the request and a descriptive User-Agent header.

C.2. Extend C.1 to also extract the tags for each quote. Group the output by tag: for each tag, list all quotes that have that tag.

C.3. Write a scraper that fetches the first 3 pages of http://quotes.toscrape.com/tag/life/ (quotes tagged "life") and collects all unique authors. Print the authors sorted alphabetically.

C.4. Write a function check_robots_txt(url: str) -> str that fetches and displays the robots.txt file for a given domain. Test it with http://quotes.toscrape.com/robots.txt. What does the file tell you about which paths are allowed or disallowed?

Part D: File Automation ⭐⭐

D.1. Write a script that scans a directory and generates a report showing: - Total number of files - Total size (in human-readable format: KB, MB, etc.) - Breakdown by file extension - The 5 most recently modified files

Test it on your own project directory.

D.2. Write a function find_duplicates(directory: Path) -> list[list[Path]] that finds files with identical sizes in a directory. (For a simple heuristic, files with the same size might be duplicates. A production version would compare file hashes, but size comparison is fine for this exercise.)

D.3. Write a script that monitors a directory for new files. Every 5 seconds, it checks whether any new files have appeared and prints a notification. Use time.sleep(5) in a loop. (Hint: compare the current set of files to the set from the previous check.)

Part E: Automation Pipelines ⭐⭐⭐

E.1. Build a mini pipeline that: 1. Reads a CSV file of student grades 2. Validates that all grades are between 0 and 100 3. Computes class statistics (mean, median, highest, lowest) 4. Generates a formatted text report 5. Saves the report to a file with a timestamped filename

Use this sample CSV data:

name,exam1,exam2,final
Alice,92,88,95
Bob,78,82,71
Carol,95,91,98
David,65,70,68
Eve,88,85,90

E.2. Create an automation script that generates a "daily digest" file every time it runs. The digest should include: - Current date and time - A quote (scraped or from a local list) - A summary of your TaskFlow tasks (read from JSON) - Weather data placeholder (print "Weather: [integrate API from Ch 21]")

Save each digest to a digests/ directory with the filename digest_YYYYMMDD.txt.

E.3. (Spaced review — Chapter 21, APIs) Modify the scraper from C.1 to save its results as both JSON and CSV files. Compare: which format is better for this data, and why? (Think about nested data like tags.)

Part F: Integration and Design ⭐⭐⭐⭐

F.1. Build a "link checker" that takes an HTML file (or string) and checks whether all the href links in <a> tags return a 200 status code. For each link, report whether it's valid, broken (4xx), or unreachable (connection error). Add rate limiting with time.sleep(1) between requests.

F.2. (Spaced review — Chapter 18, Recursion) Write a recursive function scrape_site_map(url: str, max_depth: int = 2) -> dict that, given a starting URL, finds all internal links on the page, then follows those links to find more internal links, up to max_depth levels deep. Return a nested dictionary representing the site structure. Use http://quotes.toscrape.com as your test site. Be sure to include rate limiting and avoid visiting the same URL twice.

F.3. Design (but don't necessarily implement) an automation system for Dr. Patel's genome database scraper. Write pseudocode and a design document that covers: - What data to scrape and from where - Rate limiting strategy - Error handling and retry logic - Data storage format - Scheduling approach - Ethical considerations specific to scientific data

This is a design exercise — focus on the thinking, not the code.