Exercises: API Ingestion
Exercise 16.22(c) — proving the retry classifier, especially the 400 case — is the one that prevents the incident in §16.6. Do that one.
Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper.
Solutions: daggered (†) and odd-numbered problems are in
appendices/answers-to-selected.md.
Part A — Warm-ups ⭐
16.1 † Name the five ways an API is harder to extract from than a database.
16.2 Name the four pagination shapes. Which three can lose or duplicate rows, and which is correct?
16.3 † Explain, with a concrete sequence, how offset pagination loses rows on a collection that is receiving inserts.
16.4 What is the one-parameter fix for §16.2's ⚠️ callout, and why does it work?
16.5 † Why should you terminate page-token pagination on the token's absence rather than on an empty page?
16.6 Name four rate-limit response headers and say what each means. Which one is authoritative over your local token bucket, and why?
16.7 † Classify as retryable or terminal: 400, 401, 408, 429, 500, 503, 422, a connection reset.
16.8 Name the three terminal errors that are actively harmful to retry, and what each causes.
16.9 † What is jitter for? Explain what happens without it when 200 clients fail simultaneously.
16.10 Why refresh an auth token on a margin rather than at expiry? Name the three ways people get this wrong.
Part B — Standard ⭐⭐
16.11 Compute the wall-clock cost of a backfill for three providers: (a) 10 req/s, 100 per page; (b) 1 req/s, 50 per page; (c) 100 req/s, 1000 per page. All fetching 2.9 million records. Then state which of §16.3's three mitigations you would pursue first for each.
16.12 † Implement the RateLimiter from §16.3 and test it: assert that it does not exceed the
configured rate over a 30-second run, that it obeys a simulated Retry-After, and that it slows down
when X-RateLimit-Remaining drops below the burst size.
16.13 Write the error classifier from §16.6 as a function returning RETRY, REFRESH_THEN_FAIL,
or FAIL. Then write the test suite. The 400 test matters most — assert it is never retried, and
explain in a comment what happens if it is.
16.14 † Implement TokenProvider with the refresh margin and make it thread-safe. Then write the
test that proves eight concurrent callers cause exactly one refresh — and explain what goes wrong at
a provider that revokes the previous token on issue.
16.15 §16.8 measures the re-fetch window from data. Design the instrumentation: what you store, how you detect a change, and how you produce the table. Then estimate its storage cost for Kestrel's carrier data at 2.9 million records a year.
16.16 † Write the nightly contract test from §16.9 for an API you have access to (or for a public one — a weather, transit, or open-data API). Include the asymmetry: missing or retyped fields fail, new fields log. Run it. Record what it says.
16.17 A provider offers webhooks and a polling endpoint. Design the pipeline that uses both, and specify: what the webhook handler does, what the poller does, how they avoid duplicating work, and the reconciliation that tells you the webhook is dropping events.
Part C — Deeper ⭐⭐⭐
16.18 §16.8's 📐 callout handles the long tail with a reconciliation rather than a wider window. Generalize this: find two other places in this book where a long tail is handled by a different mechanism rather than by extending the primary one. State the general principle.
16.19 † The §16.6 incident was preceded by a grace period during which 429s were returned but not enforced. Design the alerting that would have caught it. Then generalize: what other "warning shots" do APIs and databases fire that pipelines typically ignore?
16.20 Construct the case for not retrying at all — a client that fails fast on any error and relies on the orchestrator to retry the whole job. What does it gain, what does it cost, and for what kind of extraction is it right?
16.21 † §16.9 says recorded fixtures "go stale silently." Design the mechanism that detects this: how would you know your fixtures no longer resemble the API? What does it cost to run, and how does it differ from the contract test?
Part D — The Kestrel Platform ⭐⭐⭐
16.22 — Increment 16: the carrier ingester.
(a) Write platform/ingest/api/carrier.py with four components: a header-steered rate limiter, a
token provider with a refresh margin and thread safety, an error classifier, and cursor pagination
that records the last record id as well as the cursor.
(b) Land raw JSON to bronze/carrier/v1/fetch_date=.../, unparsed, with Chapter 9 §9.4's
envelope.
(c) Prove the classifier. Tests asserting: 429 waits for Retry-After; 401 refreshes once then
fails; 400 is never retried; 503 backs off with jitter. Use a stub server so the tests are fast
and deterministic.
(d) Measure your re-fetch window. Store a content hash per record. After a fortnight, produce §16.8's table for your own data and choose N from it.
(e) Write the contract test and schedule it nightly.
(f) Add a 429 counter with an alert — the control that would have given three days' warning before the ban.
16.23 † Add platform/ingest/api/replay.py: given a fetch_date range, re-parse the raw JSON in
bronze without calling the API at all.
This is the payoff for landing raw (Chapter 9 §9.4), and it is what turns "we need to re-extract six months" — 16 hours against a rate limit — into a local reprocessing job. Time both and record the difference.
Reflection
A. §16.6's incident had a client that "worked for eight months" and then got banned in an afternoon by a change on the other side. What else in your systems is working only because something outside your control has not changed yet?
B. The chapter says retry classification matters more than backoff policy. Where else does the decision about whether to act matter more than the mechanics of acting?