Chapter 21 Key Takeaways: Working with APIs and External Data Services

Core Concepts

APIs are structured data pipelines, not magic. An API call is just an HTTP request — the same protocol your browser uses — with your code making the request programmatically and processing the result. Understanding this removes the mystery. Every API call follows the same pattern: build the URL, add authentication, specify parameters, make the request, check the status code, parse the JSON.

The requests library is the standard tool. Install it once (pip install requests), import it, and you have a clean interface to any HTTP API. The essential methods are requests.get() for retrieving data and requests.post() for sending it. The essential response properties are .status_code, .json(), and .raise_for_status().

Status codes are the API's answer to "did it work?" Your code must read status codes before processing data. 200 means success. 4xx means something is wrong on your side (auth, bad params, resource not found). 5xx means something is wrong on the server side. 429 means you're asking too fast. Never assume a response is 200 unless you've checked.

Authentication

API keys belong in environment variables, never in code. A key hardcoded in a Python file will eventually end up in version control, an email, a shared drive, or a pastebin. Keys in environment variables stay in the environment. The python-dotenv library makes this easy: put keys in .env, load with load_dotenv(), read with os.environ.get("KEY_NAME"). Add .env to .gitignore on day one.

Authentication patterns vary by API. The three most common: API key in a header (most common for data APIs), API key in query parameters (some older APIs), Bearer token in the Authorization header (OAuth2 and JWT). Read the documentation for each API — there is no universal standard for the header name.

Data Handling

JSON maps directly to Python. JSON objects become dicts, arrays become lists, strings are strings, numbers are ints or floats, null is None. response.json() does this conversion automatically. You can navigate the result with the same dictionary and list syntax you use for any Python data structure.

Use .get() for nested access, not bare bracket notation. data["key"] raises KeyError if the key is missing. data.get("key", default) returns the default. Real API responses frequently have optional fields. Code that uses bare brackets on API data is a KeyError waiting to happen.

Pagination is the norm for large datasets. Any API that returns lists of records will paginate beyond a certain size. Your fetch functions should handle pagination from the beginning — add a while True loop that requests pages until it gets an empty one, and always include a max_pages guard to prevent infinite loops.

Reliability

Retry logic is not optional for production code. Networks fail, servers hiccup, rate limits are hit. An API call that worked yesterday may timeout today. The api_request_with_retry() pattern from this chapter covers the common cases: retry on 429 and 5xx, respect Retry-After headers, back off exponentially, give up after a configurable number of attempts.

Exponential backoff prevents making the problem worse. When a server is overloaded, hammering it with retries makes it more overloaded. Waiting 1s, then 2s, then 4s, then 8s between retries is polite and effective. Always cap the maximum wait time (120 seconds is reasonable).

A timeout parameter is required. Omitting timeout means a hung server can block your script indefinitely. Set timeout=30 as a reasonable default. Reduce it to 10-15 seconds for APIs you know are fast. Always set it.

Design Patterns

The APIClient class is worth building once. Rather than copy-pasting auth headers, retry logic, and error handling into every API call, encapsulate these concerns in a reusable class. Your business logic code then reads like client.get("/rates/USD") rather than 20 lines of boilerplate.

Use requests.Session for multiple calls to the same API. A Session reuses the underlying TCP connection (connection pooling), which measurably reduces latency when making many sequential calls to the same host. It also allows you to set default headers once rather than on every request.

Always validate response structure before using it. APIs change, APIs have bugs, APIs return different structures for different error conditions. After parsing JSON, check that the keys you expect are present before accessing them. This is especially important for fields you derive business calculations from.

Free APIs Worth Knowing

Business Value

APIs turn one-time analysis into a live data connection. The difference between a static CSV from last month and an API-powered dashboard that reflects today's exchange rates, today's news, and today's weather is the difference between a historical document and an operational tool.

Combining multiple data sources creates insights no single source provides. Acme's internal sales data is valuable. The competitor news from NewsAPI is valuable. The USD/CAD exchange rate from open.er-api.com is valuable. None of those alone answered Sandra Chen's question. Together, they did.

Credentials management is a professional skill, not a detail. Organizations have been breached because an API key was committed to a public GitHub repository. The habit of using environment variables protects your organization and demonstrates professional competence. Build the habit now, before you have a key worth protecting.

Common Mistakes to Avoid

• Calling response.json() without first checking response.status_code
• Using data["key"] instead of data.get("key") on API responses
• Hardcoding API keys in source code
• Omitting the timeout parameter on requests
• Writing pagination loops without a max_pages safety limit
• Retrying 401 or 400 errors (they won't fix themselves — don't waste requests)
• Ignoring Retry-After headers when handling 429 responses

Chapter 21 — Part of "Python for Business for Beginners: Coding for Every Person"