Chapter 16 — Key Takeaways (API Ingestion)
The page for writing an API client, and for the morning you get a 429.
Five ways an API is harder than a database
- You cannot see the schema — only documentation, which is frequently wrong about optional fields, nulls, and enums
- You cannot see what changed — no log, no CDC, and any modified-since filter carries all four of Ch. 13 §13.4's lies with no way to check
- You are a guest with an enforced quota — and exceeding it can mean blocked, not throttled
- Every request can fail independently — 40,000 pages at 0.1% is forty failures a run, so retry handling is the main loop, not an edge case
- You cannot test against production
Pagination — three of four shapes lose rows
| Shape | Failure |
|---|---|
| Offset | ⚠️ systematically lossy on a growing collection |
| Cursor / keyset | ✅ correct — but cursors expire; record the last record id too |
| Page token | opaque — never parse; terminate on the token's ABSENCE, not an empty page |
| Link header | follow the URL, do not build it |
page 20 read at offset 1900-1999
← 6 new records arrive; everything shifts by 6
page 21 at offset 2000-2099 → re-reads 6, and 6 slide past the last page FOREVER
Paginate over something that does not move. Ascending order on an immutable field, plus a closed upper bound so the run's scope is a decision rather than an outcome.
Rate limits
X-RateLimit-Limit / -Remaining / -Reset Retry-After: 30
Be proactive. A client that runs flat out until throttled spends its life in backoff and may get
blocked. Steer from the server's Remaining header — your local bucket does not know about the
other systems sharing your quota.
$$\text{backfill hours} = \frac{\text{records}}{\text{page size}} \div \text{req/s}$$
The rate limit, not the data volume, decides whether a backfill is an afternoon or a weekend. Mitigations by payoff: bigger page size (often one parameter, 10×) · an undocumented bulk endpoint — ask · parallelism (only if latency-bound, not quota-bound).
Authentication
Refresh on a MARGIN (5 min), not at expiry, and make it thread-safe — some providers revoke the previous token on issue, so eight concurrent refreshes give seven invalid tokens and a 401 cascade that looks like a credential problem.
Three ways people get it wrong: fetch once at startup · refresh on 401 · refresh exactly at expiry.
Backoff — and jitter is not a refinement
delay = random.uniform(0, min(cap, base * 2 ** attempt)) # FULL jitter
Without it, 200 clients that failed together retry together — backoff merely synchronizes the herd at a longer interval.
Retry-After always takes precedence over your exponential. It can be seconds OR an HTTP
date — handle both.
⚠️ Which errors to retry — this matters more than the backoff
| Retryable | Terminal |
|---|---|
408, 429 (after Retry-After), 500, 502, 503, 504, network errors |
400, 401, 403, 404, 409, 422 |
Default: unknown 4xx → FAIL, unknown 5xx → RETRY. A 4xx is about your request.
Three terminal errors that are actively harmful to retry:
| Causes | |
|---|---|
| 401 | looks like a brute-force attempt → account blocked |
429 without Retry-After |
exactly the behavior rate limits exist to stop |
| 400 | turns one malformed request into thousands |
Backoff without classification AMPLIFIES the problem. Six 429s per logical request, against one for a client with no retry logic at all. When you add a mechanism that reacts to errors, ask whether its reaction can cause the error it reacts to.
Incremental extraction
| Provider gives you | You do |
|---|---|
| A modified-since filter | use it, with an overlap window; assume all four lies |
| Immutable creation ordering | fetch new + a periodic re-fetch window for changes |
| Nothing | full extraction, or a webhook |
Webhooks are lossy in practice whatever the documentation says. The pattern is both: webhooks for latency, a periodic pull for completeness, and a reconciliation that tells you the webhook is dropping things.
Retroactive change — no database analogue
Size the re-fetch window by MEASUREMENT. Kestrel, measured over three months:
| Days after creation | Share of changes |
|---|---|
| 0–1 | 71% |
| 2–7 | 24% → 7 days = 95% |
| 8–30 | 4.6% → 30 days = 99.6% |
| 31–90 | 0.4% |
| >90 | 0.02% |
Handle the tail with a DIFFERENT mechanism, not a wider window. 90 days costs 13× the requests for a further 0.38%. A monthly reconciliation against the provider's own aggregate catches it at constant cost.
Testing — three layers
- Recorded fixtures — fast, deterministic, and they go stale silently
- A nightly contract test — one request per endpoint, asserting structure not values ← the layer that catches the API changing, and the one people skip
- A sandbox account, if offered
The asymmetry: a missing or retyped field fails; a new field logs. Ch. 13 §13.7's drift taxonomy, applied to an API.
The one metric to add today
metrics.increment("http_response", tags={"host":…, "status":…, "endpoint":…})
Alert on 429 with a threshold of ZERO. Any sustained 429 is the API explicitly telling you your behavior is a problem.
Warning shots pipelines routinely ignore because the job still succeeds:
| Signal | Ignored because |
|---|---|
| 429 | retries get through |
Deprecation / Sunset headers |
nothing breaks yet |
| PostgreSQL warnings | the query returns rows |
| Kafka rebalance log lines | throughput is merely degraded |
| Replication lag alerts | they self-resolve |
Every one has caused an incident in this book. Monitor what the system tells you, not only whether the job exited zero.
And two rules that generalize
Retry at the smallest granularity that can succeed, and make the next level up resumable. A job-level retry of a 40,000-request extract wastes 39,000 requests against a quota.
If a mechanism suppresses a symptom, you owe a measurement in its place. Idempotency owes a duplicate-rate metric · an overlap window owes a reconciliation · a DLQ owes a rate alert.