Web scraping extracts data from HTML pages using Beautiful Soup, and automation combines scraping with file handling, data transformation, and scheduling to turn hours of manual work into seconds of script execution.
The Scraping Pipeline
Stage
Tool
What It Does
Fetch
requests.get(url)
Downloads the HTML from a URL
Parse
BeautifulSoup(html, "html.parser")
Turns HTML text into a navigable tree
Extract
soup.find(), soup.select()
Locates specific elements by tag, class, ID, or CSS selector
Transform
Python (dicts, lists, loops)
Cleans and structures the extracted data
Output
pathlib, json, csv
Saves results to files or other formats
Beautiful Soup Quick Reference
Method
Returns
Use When
find("tag")
First matching Tag or None
You need one specific element
find_all("tag")
List of all matching Tag objects
You need every matching element
select("css")
List of Tag objects matching CSS selector
Complex queries (nested, multiple classes)
select_one("css")
First matching Tag or None
CSS selector for one element
tag.text
String of text content
Extract visible text from a tag
tag["attr"]
Attribute value
Get href, class, data-*, etc.
tag.get("attr")
Attribute value or None
Safely get an attribute that might not exist
CSS Selector Cheat Sheet
Selector
Meaning
Example
tag
Element by tag name
"p" matches all <p> tags
.class
Element by class
".price" matches class="price"
#id
Element by ID
"#main" matches id="main"
parent child
Descendant
".product .price" — .price inside .product
tag.class
Tag with class
"div.item" — <div> with class="item"
[attr]
Has attribute
"[data-id]" matches elements with data-id
[attr=val]
Attribute equals
"[type=email]" matches type="email"
Ethical Scraping Checklist
Step
Action
1
Check if an API exists first — use it if available
2
Read robots.txt at the site's root
3
Review the Terms of Service
4
Set a descriptive User-Agent header
5
Add time.sleep() between requests (minimum 1 second)
6
Respect Crawl-delay in robots.txt
7
Don't scrape data behind login walls
8
Store only what you need; don't redistribute without permission
Automation Patterns
Pattern
Example
Key Idea
Fetch-Parse-Transform-Output
Elena's report pipeline
Chain stages; each does one job
File organization
Sort downloads by extension
pathlib + shutil.move()
Batch processing
Rename 500 photos
Loop over Path.iterdir()
Report generation
Weekly summary to text/JSON
Build strings, save with timestamp
Scheduled execution
Run daily at 8 AM
cron (Unix) or Task Scheduler (Windows)
Scheduling Quick Reference
Platform
Tool
How to Set Up
Linux/macOS
cron
Edit with crontab -e; syntax: min hour day month weekday command
Windows
Task Scheduler
GUI or schtasks CLI
Cross-platform
schedule library
pip install schedule; runs inside Python process
Cron examples:
0 8 * * * python3 /path/to/script.py # Daily at 8:00 AM
0 8 * * 1 python3 /path/to/script.py # Every Monday at 8:00 AM
*/30 * * * * python3 /path/to/script.py # Every 30 minutes
When NOT to Scrape
Situation
Better Alternative
Site offers an API
Use the API
Data is behind a login
Request API access or data export
robots.txt disallows it
Respect the rules
Terms of Service prohibit it
Find another data source
Data is copyrighted
Check licensing; consider fair use
Small site that can't handle load
Contact the owner
Common Pitfalls
Pitfall
Symptom
Fix
No parser specified
GuessedAtParserWarning
Always pass "html.parser"
No rate limiting
IP gets blocked; server overloaded
Add time.sleep(1) between requests
No error handling
Script crashes on network errors
Wrap requests in try/except
Scraping JavaScript-rendered pages
Empty results
Use Selenium or check for an API
Hardcoded CSS selectors
Scraper breaks on site redesign
Document assumptions; add validation
TaskFlow Progress
v2.3: Daily motivational quote scraper (from quotes.toscrape.com with caching); automated task report generation to timestamped text files; fallback offline mode for quote display.
What's Next
Chapter 25: Version control with Git — track your code's history, collaborate with others, and experiment fearlessly because you can always go back.