Quiz: Web Scraping and Automation
Test your understanding before moving on. Target: 70% or higher to proceed confidently.
Section 1: Multiple Choice (1 point each)
1. What does Beautiful Soup's find_all("div", class_="item") return?
- A) The first
<div>with class "item" - B) A list of all
<div>elements with class "item" - C) A string containing all matching HTML
- D) A dictionary mapping class names to elements
Answer
**B)** A list of all `` elements with class "item"
*Why B:* `find_all()` returns a `ResultSet` (list-like) of all matching `Tag` objects. It finds *all* matches, not just the first.
*Why not A:* That's what `find()` does (singular, returns first match).
*Why not C:* It returns Tag objects, not raw strings.
*Why not D:* Beautiful Soup returns Tag objects in a list, not a dictionary.
*Reference:* Section 24.3
2. What is the purpose of robots.txt?
- A) It encrypts web traffic between the scraper and server
- B) It specifies which parts of a website crawlers should or should not access
- C) It provides an API key for automated access
- D) It lists the programming languages the website was built with
Answer
**B)** It specifies which parts of a website crawlers should or should not access *Why B:* `robots.txt` is a voluntary standard that website owners use to communicate which paths are off-limits to automated crawlers. It's a guideline, not a technical enforcement — but ethical scrapers respect it. *Why not A:* `robots.txt` has nothing to do with encryption. HTTPS handles that. *Why not C:* API keys are a separate authentication mechanism. *Why not D:* It's about crawler access rules, not technology disclosure. *Reference:* Section 24.53. Which line of code correctly creates a Beautiful Soup object from an HTML string?
- A)
soup = BeautifulSoup.parse(html) - B)
soup = BeautifulSoup(html, "html.parser") - C)
soup = BeautifulSoup(html) - D)
soup = parse_html(html)
Answer
**B)** `soup = BeautifulSoup(html, "html.parser")` *Why B:* The `BeautifulSoup` constructor takes the HTML string as the first argument and the parser name as the second. `"html.parser"` is Python's built-in parser — no extra installation needed. *Why not A:* There is no `BeautifulSoup.parse()` class method. *Why not C:* While this technically works, it raises a `GuessedAtParserWarning`. Always specify the parser explicitly. *Why not D:* `parse_html` is not a standard Beautiful Soup function. *Reference:* Section 24.34. Why should you add time.sleep() between HTTP requests in a scraper?
- A) To make the code easier to debug
- B) To avoid overwhelming the server and to be a responsible scraper
- C) Python requires a delay between network calls
- D) To ensure the HTML has time to load in the browser
Answer
**B)** To avoid overwhelming the server and to be a responsible scraper *Why B:* Sending hundreds of requests per second can overload servers, slow them down for other users, and get your IP address blocked. Rate limiting (adding delays) is a core ethical scraping practice. *Why not A:* Debugging is unrelated to inter-request delays. *Why not C:* Python has no such requirement — you *can* send requests as fast as you want. You *shouldn't*. *Why not D:* `requests.get()` fetches the fully rendered HTML from the server. Browser rendering is a different concept (relevant to headless browsers, not basic scraping). *Reference:* Section 24.55. What is the difference between soup.find() and soup.select_one()?
- A)
find()returns a string;select_one()returns a Tag - B)
find()uses tag names and attributes;select_one()uses CSS selector syntax - C)
find()searches the entire document;select_one()only searches the first level - D) They are identical — two names for the same function
Answer
**B)** `find()` uses tag names and attributes; `select_one()` uses CSS selector syntax *Why B:* `find("div", class_="item")` is Beautiful Soup's native API. `select_one("div.item")` uses CSS selector strings (the same selectors used in web development). Both return a single Tag or None. The selector syntax is more concise for complex queries. *Why not A:* Both return `Tag` objects (or `None`). *Why not C:* Both search the entire document tree recursively. *Why not D:* They use different query syntaxes, even if results are often the same. *Reference:* Section 24.36. Which of the following is the best reason to use an API instead of scraping?
- A) APIs are always faster than scraping
- B) APIs provide structured data with a stable interface, while scraped HTML can change without warning
- C) Scraping is always illegal
- D) APIs don't require any authentication
Answer
**B)** APIs provide structured data with a stable interface, while scraped HTML can change without warning *Why B:* APIs return clean JSON/XML with documented fields. Scraped HTML depends on the page's visual structure, which can change any time the site redesigns — breaking your scraper. APIs also have versioning and documentation. *Why not A:* APIs aren't necessarily faster; sometimes scraping gets data more quickly than waiting for API pagination. *Why not C:* Scraping is not inherently illegal (though it can violate terms of service in some cases). *Why not D:* Many APIs require API keys or OAuth authentication. *Reference:* Section 24.97. What does response.raise_for_status() do in the requests library?
- A) Prints the HTTP status code to the console
- B) Raises an
HTTPErrorexception if the status code indicates failure (4xx or 5xx) - C) Retries the request if it failed
- D) Returns
Trueif the status is 200
Answer
**B)** Raises an `HTTPError` exception if the status code indicates failure (4xx or 5xx) *Why B:* `raise_for_status()` checks the response's status code. If it's in the 200-399 range, it does nothing. If it's 400 or above (client or server error), it raises a `requests.exceptions.HTTPError`, which you can catch with a `try/except` block. *Why not A:* It doesn't print anything — it raises an exception. *Why not C:* It doesn't retry. Retry logic must be implemented separately. *Why not D:* It returns `None` on success, not a boolean. *Reference:* Section 24.48. On macOS/Linux, which tool is used to schedule a script to run at regular intervals?
- A) Task Scheduler
- B)
cron - C)
schedule.every() - D)
time.sleep()in a loop
Answer
**B)** `cron` *Why B:* `cron` is the standard Unix/Linux/macOS job scheduler. You configure it by editing the crontab with `crontab -e` and specifying when to run each command. *Why not A:* Task Scheduler is the Windows equivalent. *Why not C:* `schedule` is a Python library that provides in-process scheduling, but it requires the Python script to stay running. `cron` runs at the OS level. *Why not D:* A `time.sleep()` loop requires the script to run continuously, which is fragile and wastes resources. *Reference:* Section 24.7Section 2: Short Answer (2 points each)
9. Explain the difference between soup.find("p") and soup.find_all("p"). When would you use each?
Answer
`find("p")` returns the **first** `` tag found in the document (or `None` if none exist). `find_all("p")` returns a **list** of all `
` tags in the document (empty list if none exist). Use `find()` when you know there's only one element you need (like a page title or a specific container). Use `find_all()` when you need to process multiple matching elements (like all paragraphs, all list items, or all product entries on a page). *Reference:* Section 24.3
10. A website's robots.txt contains:
User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 5
What does this tell your scraper? List three specific things you should do in response.
Answer
This `robots.txt` tells us: 1. **All crawlers** (`User-agent: *`) are subject to these rules. 2. **Do not access** any URLs under `/admin/` or `/private/`. 3. **Wait at least 5 seconds** between requests (`Crawl-delay: 5`). Three things to do: 1. Never request any URL starting with `/admin/` or `/private/`. 2. Set `time.sleep(5)` (or more) between consecutive requests to this domain. 3. Identify your scraper with a descriptive `User-Agent` header so the site owner can contact you if there's an issue. *Reference:* Section 24.511. Why is "html.parser" passed as the second argument to BeautifulSoup()? What happens if you omit it?
Answer
The second argument specifies which **parser** Beautiful Soup should use to interpret the HTML. `"html.parser"` is Python's built-in parser (no extra installation required). Other options include `"lxml"` (faster, requires separate installation) and `"html5lib"` (most lenient, also requires installation). If you omit the parser argument, Beautiful Soup will try to guess which parser to use and will print a `GuessedAtParserWarning`. The behavior may differ across systems depending on which parsers are installed, making your code less portable. Always specify the parser explicitly. *Reference:* Section 24.312. Describe three scenarios where you should NOT scrape a website and should use a different approach instead.
Answer
1. **The site offers a public API.** APIs provide structured, stable data and are the intended way for programs to access the site's content. Scraping when an API exists is wasteful and fragile. 2. **The site's `robots.txt` disallows your target paths** or the Terms of Service explicitly prohibit scraping. Respecting these rules is both ethical and can have legal implications. 3. **The data is behind a login wall or paywall.** Scraping content that requires authentication to access raises serious legal and ethical concerns — even if you have an account, automated extraction may violate the terms of service. Other valid answers: the data is copyrighted and redistribution is not allowed; the data involves personal/private information; the site is small and your scraper could disrupt service for other users. *Reference:* Section 24.9Section 3: Code Analysis (3 points each)
13. What does the following code print? Trace through it carefully.
from bs4 import BeautifulSoup
html = """
<div class="results">
<p class="score">85</p>
<p class="score">92</p>
<p class="score">78</p>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
scores = soup.select(".score")
total = sum(int(s.text) for s in scores)
print(f"Average: {total / len(scores):.1f}")
Answer
**Output:** `Average: 85.0` Step by step: 1. `soup.select(".score")` finds all elements with class "score" — three `` tags. 2. The generator expression `int(s.text) for s in scores` extracts the text from each tag and converts to int: 85, 92, 78. 3. `sum(...)` adds them: 85 + 92 + 78 = 255. 4. `len(scores)` is 3. 5. `255 / 3 = 85.0`, formatted to one decimal place. *Reference:* Section 24.3
14. Find the bug in this scraper function:
import requests
from bs4 import BeautifulSoup
def scrape_titles(url):
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
titles = soup.find_all("h2")
for title in titles:
print(title)
List at least three problems with this code.
Answer
1. **No error handling.** If the request fails (network error, 404, timeout), the code crashes. Should use `try/except` and `response.raise_for_status()`. 2. **No rate limiting.** No `time.sleep()` before the request. If called in a loop, this would hammer the server. 3. **No User-Agent header.** The default `requests` User-Agent (`python-requests/2.x`) is often blocked by websites. Should include a descriptive custom header. 4. **Prints the entire tag, not just the text.** `print(title)` prints `Some Title
` including HTML tags. Should use `print(title.text)` to get just the text content. 5. **No timeout.** `requests.get(url)` without a `timeout` parameter can hang indefinitely if the server doesn't respond. *Reference:* Sections 24.4, 24.515. What does this automation script do? Describe each stage.
from pathlib import Path
from datetime import datetime
def cleanup_logs(log_dir: Path, days_old: int = 30) -> int:
cutoff = datetime.now().timestamp() - (days_old * 86400)
removed = 0
for log_file in log_dir.glob("*.log"):
if log_file.stat().st_mtime < cutoff:
log_file.unlink()
removed += 1
return removed
Answer
This function **automatically deletes old log files** from a directory. 1. **Calculate cutoff time:** `days_old * 86400` converts days to seconds. Subtracting from the current timestamp gives a cutoff — any file modified before this time is "too old." 2. **Find log files:** `log_dir.glob("*.log")` finds all files ending in `.log` in the specified directory. 3. **Check modification time:** `log_file.stat().st_mtime` gets the file's last modification time as a Unix timestamp. If it's older than the cutoff, the file is deleted. 4. **Delete and count:** `log_file.unlink()` deletes the file. The function counts and returns how many files were removed. This is a good example of automation beyond scraping — routine file maintenance that would be tedious to do manually. *Reference:* Section 24.6Section 4: Scenario-Based (3 points each)
16. You need to collect the titles and prices of all books on a website. The site has 50 pages of results. Describe your scraping strategy, including how you would: - Handle pagination - Be a responsible scraper - Store the results - Handle errors
Answer
**Pagination:** Fetch page 1, find the "next page" link, follow it, repeat until no more pages or reaching page 50. Alternatively, if the URL pattern is predictable (`/page/1`, `/page/2`, etc.), iterate through page numbers directly. **Responsible scraping:** - Check `robots.txt` before starting. - Add `time.sleep(1)` or more between page requests. - Use a descriptive `User-Agent` header. - Limit the total number of requests per session. - Consider if the site has an API instead. **Storage:** Collect results in a list of dictionaries, then save as JSON (preserves structure) or CSV (if flat data — titles and prices are flat). Include a timestamp for when the data was collected. **Error handling:** - Wrap `requests.get()` in `try/except` to handle network errors. - Use `response.raise_for_status()` to catch HTTP errors (404, 500, etc.). - If a page fails, log the error and continue to the next page rather than crashing. - Validate extracted data (e.g., prices should be numeric). *Reference:* Sections 24.4, 24.5, 24.817. Your friend says: "I wrote a scraper that checks a competitor's prices every 30 seconds so my store can always undercut them by $0.01." Evaluate this from both a technical and ethical standpoint.
Answer
**Technical problems:** - 30-second intervals is extremely aggressive. The friend's IP will likely get blocked quickly. Even 1-minute intervals are considered fast for most sites. - No mention of `robots.txt` compliance. - Price pages often require JavaScript rendering, so `requests + Beautiful Soup` may not work — might need Selenium or a headless browser, which are much slower. - HTML structure changes will break the scraper with no warning. - Running 24/7 at this rate amounts to ~2,880 requests/day to a single site. **Ethical problems:** - Likely violates the competitor's Terms of Service. - Could constitute unfair business practices or even be illegal under some jurisdictions' computer fraud laws. - The aggressive rate could degrade the competitor's site performance, affecting their legitimate customers. - Automated price undercutting raises antitrust/anti-competitive concerns. - The competitor almost certainly has an expectation that their pricing isn't being machine-monitored at this frequency. **Better approach:** Check if the competitor offers a price API, use authorized price comparison services, or if scraping is truly necessary, do it once or twice a day with proper rate limiting and legal review. *Reference:* Sections 24.5, 24.9Section 5: Spaced Review (1 point each)
18. (Chapter 18 — Recursion) How could recursion be useful in web scraping? Give a specific example of a scraping task that is naturally recursive.
Answer
Web scraping is naturally recursive when following links to build a **site map** or **crawling nested pages**. For example, scraping a documentation site: start at the index page, find all section links, follow each link, find sub-section links within each section, and so on. Example: A function `crawl(url, depth)` that scrapes a page, extracts all internal links, and recursively calls `crawl(link, depth - 1)` for each link up to a maximum depth. The base case is `depth == 0` or no unvisited links remaining. This mirrors the recursive structure of a tree: the website's page hierarchy is a tree, and crawling it is a tree traversal — exactly the kind of problem recursion excels at ([Chapter 18](../../part-06-algorithms-and-data-structures/chapter-18-recursion/index.md)). *Reference:* [Chapter 18](../../part-06-algorithms-and-data-structures/chapter-18-recursion/index.md) (Recursion), Section 24.419. (Chapter 21 — APIs) Compare scraping a website for weather data versus using a weather API. Give two advantages of each approach.
Answer
**API advantages:** 1. Returns clean, structured JSON data — no parsing HTML required. 2. Stable interface — the API won't break when the website redesigns. API changes are versioned and documented. **Scraping advantages:** 1. No API key required — some weather sites show data publicly but don't offer free API access. 2. Can get exactly the data you see on the page, which might include visual elements or formatting not available through the API. In practice, the API is almost always the better choice for weather data because multiple free weather APIs exist (OpenWeatherMap, wttr.in, etc.) and provide more reliable, structured data. *Reference:* [Chapter 21](../chapter-21-working-with-data/index.md) (Working with Data), Section 24.920. (Chapter 22 — Regex) You're scraping a page and need to extract all email addresses from the text. Would you use Beautiful Soup, regex, or both? Explain.