Chapter 16: API Ingestion

"The API worked perfectly for eight months. Then they added a rate limit, and we found out by being banned for a day."

Overview

Extracting from an HTTP API is the least respected skill in data engineering and one of the most universally required. Every platform has a handful of these: a payments processor, a carrier, a marketing tool, a support desk, a CRM.

The code looks trivial. GET /orders?page=1, parse the JSON, repeat. A working version takes an hour.

And an API is harder to extract from than a database in five specific ways, none of which is obvious until it costs you a day:

• You cannot see the data model, only the endpoints somebody chose to expose.
• You are rate limited, often by a rule that is undocumented, and exceeding it can get you blocked rather than throttled.
• Pagination is inconsistent, sometimes within one API, and three of its four shapes have a failure mode that loses or duplicates rows.
• Authentication expires, usually mid-job, usually on the longest job.
• The data changes retroactively. A record you read on Monday can be different on Tuesday with no indication that it changed — and unlike a database, there is no log to consult.

This chapter is those five, plus the retry discipline that everything else depends on. §16.6's distinction between a retryable and a terminal error matters more than any backoff policy, because retrying a terminal error is not merely useless — it is how you get banned.

In this chapter, you will learn to:

• Name the five ways an API is harder than a database.
• Recognize the four pagination shapes and the failure each one has.
• Read a rate limit from headers and implement a limiter that respects it.
• Classify an error as retryable or terminal.
• Implement backoff with jitter, and explain what the jitter is for.
• Handle a token that expires mid-job.
• Design incremental extraction with no reliable change signal.
• Detect retroactive changes.

Who needs this chapter: everyone. Every platform has APIs, and the failures are the same across all of them.

16.1 Five Ways an API Is Harder

1. You cannot see the schema. A database has information_schema. An API has documentation, which is a description of what someone intended, written at some point, and frequently wrong about optional fields, null behavior, and enumerated values. The only reliable schema is the one you infer from responses, which is why §16.9's contract test matters.

2. You cannot see what changed. No updated_at you can trust, no log, no CDC. Some APIs offer a modified-since filter; many do not; and several offer one that is subtly wrong (Chapter 13 §13.4's four lies apply, and you cannot inspect the source to check).

3. You are a guest with a quota. Chapter 7 §7.1's guest relationship, made explicit and enforced. Exceeding the quota can mean throttling, or a temporary block, or — with some providers — a permanent one requiring a support ticket.

4. Every request can fail independently. A database query succeeds or fails once; a 40,000-page extraction has 40,000 chances. At a 0.1% failure rate that is forty failures per run, so retry handling is not an edge case — it is the main loop.

5. You cannot test against production. You have one account, shared with the systems that use the API for real. A test that hammers it affects them.

16.2 The Four Pagination Shapes

Three of the four can lose or duplicate rows, and knowing which is which is the point.

1. Offset pagination

GET /orders?limit=100&offset=0
GET /orders?limit=100&offset=100

Simple, universal, and broken for anything that changes while you read it.

page 1: offset 0-99      you read orders 1000..901 (newest first)
        ← a new order arrives; everything shifts down by one
page 2: offset 100-199   you read orders 900..801
                         ↑ order 901 has shifted to position 100 and
                           you read it AGAIN; order 800 has shifted
                           past the boundary and you MISS it.

Inserts during pagination cause duplicates; deletes cause skips. For a table sorted newest-first and receiving inserts — which is every event-like endpoint — offset pagination is systematically lossy.

Mitigations: sort by an immutable key ascending rather than by recency; or filter to a closed time range so the result set cannot change; or use a different shape.

2. Cursor / keyset pagination

GET /orders?limit=100&after=ord_8841

The correct shape. The cursor names a position in a stable ordering, so inserts and deletes elsewhere do not shift it.

Its failure mode is cursor expiry. Many APIs invalidate a cursor after minutes to hours, so a long extraction with a pause in the middle resumes into a 400. Record the last record id as well as the cursor, so you can restart from a filter rather than from a dead cursor.

3. Page-token pagination

GET /orders?pageSize=100
→ {"items": [...], "nextPageToken": "CigKJj..."}
GET /orders?pageSize=100&pageToken=CigKJj...

Cursor pagination with an opaque token. Never construct, parse, or persist assumptions about a token — it is the provider's internal state and its format will change without notice.

Terminate on the token's absence, not on an empty page. Several APIs return a final page with items and no token; others return an empty page with a token. The token's absence is the documented signal and the empty page is not.

4. Link-header pagination

Link: <https://api.example.com/orders?page=2>; rel="next",
      <https://api.example.com/orders?page=99>; rel="last"

Follow the URL; do not build it. The provider may change parameters, hosts, or add signing, and a client that reconstructs the next URL breaks when they do. Parsing the header correctly — it is comma-separated with quoted parameters — is the only work.

⚠️ Failure Mode — Offset pagination on a growing collection

Kestrel's carrier ingester read tracking events with offset pagination, sorted newest-first, hourly. It lost data every single run and nobody noticed for five months.

The arithmetic is brutal and simple. During a 40-page extraction taking about 90 seconds, the carrier receives roughly 12 new events. Every one shifts the whole collection by one position, so on average about 12 records slide across page boundaries and are missed, and a similar number are read twice.

The duplicates were invisible because the write was idempotent — which is the correct design and which also removed the only symptom that would have been noticed.

The losses were invisible because 12 records an hour out of ~4,000 is 0.3%, and nothing was reconciling shipment events against shipments.

The fix was one parameter: sort=created_at&order=asc plus a created_after filter. Ascending order on an immutable field means new records arrive at the end, past the region you have already read, and nothing shifts.

The general rule: paginate over something that does not move. An ascending sort on an immutable key is stable; a descending sort on anything, or any sort on a mutable field, is not.

16.3 Rate Limits

Reading the limit

Most APIs publish their limits in response headers, and the headers are more current than the documentation:

X-RateLimit-Limit: 1000          # requests per window
X-RateLimit-Remaining: 847       # left in this window
X-RateLimit-Reset: 1764340800    # when the window resets (unix seconds)
Retry-After: 30                  # on a 429: wait this long

The naming varies — RateLimit-* without the X-, ratelimit-reset as seconds remaining rather than a timestamp, GitHub's X-RateLimit-Used. Read yours, and log them on every response during development, because the documentation is frequently wrong about which form you get.

Respecting it proactively

Do not rely on 429 responses to tell you. A client that runs at full speed until it is throttled is a client that spends its life in backoff, and with some providers it is a client that gets blocked.

class RateLimiter:
    """Token bucket, steered by the API's own headers.

    Two properties that matter:
      * it is PROACTIVE -- it slows down before being told to, using the
        Remaining header, rather than sprinting into a 429
      * it obeys Retry-After when it does get one
    """
    def __init__(self, requests_per_second: float, burst: int = 10):
        self.rate = requests_per_second
        self.burst = burst
        self.tokens = float(burst)
        self.updated = time.monotonic()      # monotonic: Ch. 4 §4.6
        self.reset_at: float | None = None

    def acquire(self) -> None:
        if self.reset_at and time.monotonic() < self.reset_at:
            time.sleep(self.reset_at - time.monotonic())
            self.reset_at = None
        now = time.monotonic()
        self.tokens = min(self.burst,
                          self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        if self.tokens < 1:
            time.sleep((1 - self.tokens) / self.rate)
            self.tokens = 0
        else:
            self.tokens -= 1

    def observe(self, response) -> None:
        """Steer from the server's own accounting, which is authoritative in a
        way our local count is not -- other clients share this quota."""
        remaining = response.headers.get("X-RateLimit-Remaining")
        reset = response.headers.get("X-RateLimit-Reset")
        if remaining is not None and int(remaining) < self.burst:
            # Nearly out. Wait for the window rather than racing to the wall.
            if reset:
                self.reset_at = time.monotonic() + max(
                    0, int(reset) - time.time())
        if response.status_code == 429:
            wait = int(response.headers.get("Retry-After", "60"))
            self.reset_at = time.monotonic() + wait

The observe method is what distinguishes a good client. Your local token bucket does not know about the other systems in your company sharing the same quota; the server's Remaining header does.

💸 Cost Check — What a rate limit costs in wall-clock time

Kestrel's carrier API allows 10 requests/second. A full backfill of one year of tracking events is about 2.9 million records at 100 per page:

$$\frac{2{,}900{,}000}{100} = 29{,}000 \text{ requests} \div 10/\text{s} = 2{,}900 \text{ s} = 48 \text{ minutes}$$

Comfortable. Now the same for a provider allowing 1 request/second and 50 per page:

$$\frac{2{,}900{,}000}{50} = 58{,}000 \text{ requests} \div 1/\text{s} = 16.1 \text{ hours}$$

The rate limit, not the data volume, is the constraint — and it is the number that decides whether a backfill is an afternoon or a weekend.

Three things that change the arithmetic, in order of payoff:

1. A larger page size. 50 → 500 is a 10× reduction in requests, and it is usually one parameter. Check the maximum before anything else.
2. A bulk or export endpoint. Many APIs have one, it is undocumented or buried, and it turns 58,000 requests into one job. Ask.
3. Parallelism, carefully. Concurrent requests share the same quota, so parallelism does not raise the ceiling — it only helps if you are latency-bound rather than quota-bound.

Compute this before you promise a backfill date. It is two lines of arithmetic and it is the single most common source of a missed estimate in API work.

16.4 Authentication

Four schemes, in ascending order of operational complexity.

API key in a header. Simple, static, and it will be rotated eventually. Read it from the environment; never from code.

Basic auth. Same, with base64.

OAuth 2 client credentials. Exchange a client id and secret for an access token with an expiry — typically 3,600 seconds — and refresh before it expires.

OAuth 2 authorization code with a refresh token. A user authorized once; you hold a refresh token and exchange it for access tokens. The refresh token itself can expire or be revoked, and when it does, a human must re-authorize. Plan for that: it is not an error you can retry.

⚠️ Failure Mode — The token that expired mid-job

A 90-minute backfill with a 60-minute access token. At minute 61 every request returns 401, and the job fails 68% complete.

Three ways to get this wrong, all of which happen:

1. Fetch the token once at startup. The obvious implementation, and it fails on any job longer than the token's life.

2. Refresh on 401. Better, and it wastes a request and — if the retry logic treats 401 as retryable without refreshing — produces a retry storm against an endpoint that will keep returning 401 (§16.6).

3. Refresh at expiry. Still wrong: clock skew, a slow request, and the round trip mean a token valid when you checked can be expired when the server evaluates it.

The correct version refreshes on a margin:

```python class TokenProvider: REFRESH_MARGIN = 300 # 5 minutes before expiry

def token(self) -> str:
    if (self._token is None
            or time.time() > self._expires_at - self.REFRESH_MARGIN):
        self._fetch()      # and this must be thread-safe if you paginate
    return self._token     # concurrently -- otherwise N threads each
                           # refresh, and some providers revoke the
                           # previous token when a new one is issued

```

The thread-safety note is not hypothetical. With some providers, issuing a new token invalidates the old one — so eight concurrent workers each refreshing produces seven invalid tokens and a cascade of 401s that looks exactly like a credential problem.

16.5 Backoff and Jitter

Chapter 4 §4.7's thundering herd, in code.

def backoff_delay(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
    """Exponential backoff with FULL JITTER.

    The jitter is not a refinement. Without it, N clients that failed at the
    same moment retry at the same moment -- backoff merely synchronizes the
    herd at a longer interval. With full jitter they spread uniformly.
    """
    exponential = min(cap, base * (2 ** attempt))
    return random.uniform(0, exponential)

   attempt   no jitter          full jitter
      0        1.0 s            uniform(0, 1)
      1        2.0 s            uniform(0, 2)
      2        4.0 s            uniform(0, 4)
      3        8.0 s            uniform(0, 8)

   Without jitter, 200 clients that failed together retry together, at 1s,
   then together at 2s, then together at 4s. The recovering service is hit by
   200 simultaneous requests four times instead of once.

Full jitter — a uniform draw from zero to the exponential — is the variant that performs best in practice, and it is what AWS's own guidance recommends. "Equal jitter" (half fixed, half random) is a common alternative and is slightly worse under contention.

And always respect Retry-After over your own calculation. The server knows when it will be ready; your exponential does not.

16.6 Which Errors to Retry

This matters more than the backoff policy, and it is where most API clients are wrong.

Retrying a terminal error is not merely useless. Three of them are actively harmful:

Retrying 401 hammers an auth endpoint with credentials that will not work, and several providers treat that as a brute-force attempt and block the account.

Retrying 429 without honoring Retry-After is the definition of the behavior rate limits exist to stop, and it is the fastest route to a block.

Retrying 400 on every page of a paginated extract turns one malformed request into thousands.

🏭 From the Pipeline — Banned for a day by a retry loop

A client retried every non-2xx response, five times, with exponential backoff. Reasonable-looking code, and it had run for eight months.

The provider then introduced a rate limit. The client hit 429, retried — without reading Retry-After — hit 429 again, retried, and so on. Five retries per request, across a paginated extract, against an endpoint that was telling it to stop.

The provider's abuse detection blocked the API key for 24 hours. Not throttled — blocked. Every pipeline depending on that carrier stopped, and restoring access required a support ticket and an explanation.

Three lessons:

• Classify errors before you retry them. A blanket retry is not a safe default; it is a hazard wearing a safe-looking shape.
• Honor Retry-After. It is the server telling you exactly what it wants, and ignoring it is what abuse detection is looking for.
• An API that worked for eight months can change. The client had no alerting on 429 rates, so the first signal was a total outage rather than a rising error rate.

A 429 counter with an alert would have given three days of warning, because the limit was introduced with a grace period during which 429s were returned but not enforced. Nobody was counting.

🔁 Idempotency Check — a retry is only safe if the server thinks so

Every retry in §16.5 assumes the request can be sent twice. For a GET that is free. For anything that changes state on the other side it is a promise the server has to make, and most of the time nobody has checked whether it does.

The four verbs, honestly:

text GET, HEAD idempotent by specification. Retry freely. PUT, DELETE idempotent by specification. Retry -- and read the docs anyway. POST NOT idempotent. A retry may create a second resource. PATCH depends entirely on the body. `{"count": 5}` is safe; `{"count": {"$inc": 1}}` is not, and both are PATCH.

The dangerous case is the timeout, because it is the one case where you do not know what happened. A POST that times out at 30 seconds may have been received, processed, and committed — the response is what got lost. Retrying creates a duplicate; not retrying loses the write. There is no correct answer available from the client alone.

The mechanism that resolves it is an idempotency key, and every serious API offers one:

python resp = client.post("/v1/shipments", json=payload, headers={"Idempotency-Key": shipment_uuid}) # STABLE across retries

The key must be generated once, before the first attempt, and reused by every retry — including retries in a later run of the job, which means it has to be derived from the data rather than minted at call time. uuid4() inside the retry loop is the bug, and it looks exactly like the fix.

And for extraction specifically: the ingester's own writes must be idempotent too. Kestrel's carrier ingester keys bronze rows on (carrier, tracking_number, response_hash), so a re-run over the same window produces the same rows — which is what makes §16.8's re-fetch window safe to widen. A re-fetch that duplicated rows would make the correct answer to §16.8 "as narrow as possible," and it is not.

16.7 Incremental Extraction Without a Change Feed

Most APIs give you one of three things, in descending order of usefulness.

1. A modified-since filter. GET /orders?updated_after=2025-11-28T00:00:00Z. Use it — and Chapter 13 §13.4's four lies all apply and you cannot inspect the source to check which. Use an overlap window, always, and make the write idempotent.

2. An immutable created-at ordering. No update filter, but records are ordered by creation and creation is immutable. You can incrementally fetch new records and you cannot detect changes to old ones. Combine with a periodic re-fetch window — re-read the last N days in full on a schedule — sized by how long records realistically change for.

3. Nothing. No filters, no reliable ordering. You have two options: full extraction on a schedule, if the volume permits; or a webhook, if the provider offers one.

Webhooks

The provider pushes to you. Genuinely better for latency and it introduces its own problems:

• You must be available. A missed delivery may or may not be retried, and the retry policy is the provider's.
• Delivery is at-least-once at best, and out of order.
• You must verify the signature, or anyone can post events to your endpoint.
• You still need a reconciliation path, because webhooks are lossy in practice regardless of what the documentation promises.

The standard pattern is both: webhooks for latency, a periodic full or windowed pull for completeness. Chapter 25's reconciliation is what tells you the webhook is dropping things.

16.8 Data That Changes Retroactively

The failure that has no database analogue and that people meet unprepared.

A record you read on Monday can be different on Tuesday, with no change signal. Common causes: a payment settles and its amount changes; a carrier corrects a delivery timestamp; a support ticket is merged into another; a currency conversion is restated at a month-end rate.

Your extraction read the Monday value. Nothing tells you it changed.

Three approaches:

1. A re-fetch window. Re-read the last N days in full on a schedule. Simple, effective, and it costs N days of requests per run. Size N by measurement, not by guess: track how long after creation records actually change.

2. Content hashing. Store a hash of each record; on re-fetch, compare. Detects change cheaply once you have re-fetched, and it does not tell you what to re-fetch — so it composes with the window rather than replacing it.

3. Reconciliation against a total. If the provider exposes an aggregate — a daily settlement total, a count — compare yours against it. This is the only approach that detects a change outside your re-fetch window, and it is the one to have.

📐 Design Decision — How wide should the re-fetch window be?

Kestrel's carrier data: tracking events are corrected, and the question is how far back to re-read.

They measured rather than guessing. Over three months, with content hashes on every record:

Days after creation	Share of all changes
0–1	71%
2–7	24%
8–30	4.6%
31–90	0.4%
> 90	0.02%

A 7-day window catches 95%. A 30-day window catches 99.6%. A 90-day window catches 99.98% and costs thirteen times the requests of the 7-day one.

Kestrel chose 30 days, plus a monthly reconciliation against the carrier's own monthly summary to catch the remaining 0.4%.

The reasoning worth copying: the window is sized by measurement, and the tail is handled by a different mechanism rather than by a wider window. Widening a window to catch a long tail is expensive and never complete; a reconciliation catches the tail at constant cost.

What this gives up: changes in the 0.4% band are detected monthly rather than daily. Accepted, and written down, so that when someone asks why a figure moved after month-end there is an answer.

16.9 Testing Against an API You Do Not Control

You cannot hammer production, and mocks lie.

Three layers, and you need all three:

1. Unit tests against recorded fixtures. Record real responses once (VCR-style), replay them in tests. Fast, deterministic, and they go stale silently — the API changes and your fixtures do not, so your tests keep passing against a reality that no longer exists.

Two habits keep fixtures honest, and they cost almost nothing. Date the recording — a recorded_at field in the fixture, and a test that warns when it is older than ninety days, so staleness is visible rather than assumed away. And record the failures, not only the successes. Most fixture sets contain a happy-path 200 and nothing else, which means the retry classifier, the backoff, the token refresh, and the dead-letter path — every line that runs at 3 a.m. — is the code with no test coverage at all.

2. A contract test against the real API, on a schedule. One request per endpoint, nightly, asserting the shape: required fields present, types as expected, pagination signal present. Not correctness of values — structure.

This is the layer that catches the API changing, and it is the one people skip. It costs one request per endpoint per day.

3. A sandbox account, if the provider offers one. Frequently behind on features and differently behaved, and still better than nothing for exercising error paths.

# The contract test. Nightly, one request per endpoint.
def test_tracking_endpoint_contract():
    r = client.get("/tracking", params={"limit": 1})
    assert r.status_code == 200

    body = r.json()
    assert "items" in body and "nextPageToken" in body   # pagination signal

    item = body["items"][0]
    for field, kind in EXPECTED_SHAPE.items():
        assert field in item, f"field {field!r} disappeared from the response"
        assert isinstance(item[field], kind), \
            f"{field!r} changed type: expected {kind}, got {type(item[field])}"

    # Fields we do NOT expect are informational, not failures -- an API adding
    # a field is normal and must not break the check (Ch. 13 §13.7).
    for field in set(item) - set(EXPECTED_SHAPE):
        log.info("new field in tracking response: %s", field)

Note what fails and what merely logs. A missing or retyped field fails; a new field logs. That asymmetry is Chapter 13 §13.7's schema-drift taxonomy applied to an API, and it is what keeps the test from crying wolf on every provider release.

🧭 Version Note — the deprecation you were told about in a header

APIs you do not control are versioned by someone who does, and the notice arrives somewhere your client is not looking.

http HTTP/1.1 200 OK Deprecation: Sun, 01 Mar 2026 00:00:00 GMT Sunset: Wed, 01 Jul 2026 00:00:00 GMT Link: <https://api.carrier.example/v3/docs>; rel="successor-version"

Those are standard headers (RFC 8594 for Sunset, RFC 9745 for Deprecation) and almost nobody logs them. The job keeps working, returns 200 for four months, and then returns 410 on a Wednesday — at which point the deprecation notice you were sent 120 times a day becomes very interesting.

Two lines in the client fix this permanently:

python for h in ("Deprecation", "Sunset"): if h in resp.headers: log.warning("api_deprecation", api=base_url, header=h, value=resp.headers[h])

Then alert on the log line, not on the failure. This is Chapter 25's principle applied to a dependency you do not own: the signal exists and is being discarded, which is a strictly better position than not having one.

And the same logic applies to version pinning. Pin the API version explicitly — in the URL or in an Accept header — rather than letting the vendor's default float. An unpinned client upgrades itself, at a time chosen by someone who has never heard of your pipeline, and the change arrives as a schema change with no deploy attached to it (Chapter 17 §17.8).

🧪 Try It — get yourself banned, in a sandbox

bash cd part-03-ingestion/chapter-16-api-ingestion/code python http_client.py --self-check

The self-check exercises four components against a fake server, and each one exists because of a specific incident in this chapter. Read the assertions before the code and predict each outcome:

1. Error classification. Which status codes retry, which fail immediately, and which one is a judgment call? 429 and 503 retry; 400, 401, and 404 must not — and §16.6's case study is a client that retried all of them and was blocked for 24 hours.
2. Proactive rate limiting. The client slows down from the server's headers rather than from a local bucket, because the local bucket does not know about your other three jobs.
3. Backoff with jitter. Run the retry schedule twice and diff the delays. They differ, and that is the feature — §16.5's thundering herd.
4. The circuit breaker. After N consecutive failures it stops trying. Find the state machine and say what opens it, what half-opens it, and what closes it, because the half-open state is where these are usually written wrong.

Then break one deliberately: make the classifier retry 400 and watch the request count. One malformed request becomes hundreds, which is the shape of every API-ban incident there is.

🎓 Interview Angle — "how would you ingest from a third-party API?"

A deceptively open question, and the answer people give is a library name. requests, or the vendor's SDK, and then a pause.

The strong answer is a list of things that will go wrong, because that is what the question is testing:

"The API is the easy part — the hard parts are pagination, rate limits, auth, and the fact that I can't ask it what changed. So: cursor pagination if they offer it, because offset pagination over a collection that's still being written skips rows. Backoff with jitter and a circuit breaker, and classify errors so I'm not retrying a 400. Token refresh before expiry rather than on the 401, because a long job will cross the boundary. And since there's no change feed, I'd pull by updated-at with a deliberate overlap window and make the write idempotent so the overlap is free."

The follow-up is almost always "how do you test it?" §16.9's answer: recorded fixtures for the shapes you have seen, a fake server for the shapes you have not — and the fixtures must include the error responses, because those are the paths that actually run at 3 a.m. and the ones nobody has exercised.

The question behind the question is whether you have been paged by one of these. Naming a specific failure — an offset page that skipped rows, a token that expired at hour four — does more than any list.

📏 Scale Note — the rate limit is the ceiling, and it does not move

Every other data source in this book scales by adding compute. An API does not, and that single fact determines everything about how an API ingester is designed.

text records to fetch at 3 req/s, 100 records/page wall clock ──────────────────────────────────────────────────────────────── 10,000 34 requests 11 s 100,000 334 requests 1.9 m 1,000,000 3,334 requests 18.5 m 10,000,000 33,334 requests 3.1 h 100,000,000 333,334 requests 30.9 h <-- and nothing you own makes it faster

Three design consequences follow, and they are the reason Chapter 16 looks different from Chapters 13 to 15.

A full refresh stops being possible far earlier than elsewhere. At 3 requests a second, a million records is eighteen minutes and ten million is three hours — so the incremental strategy is forced at a volume where a database would still be doing full loads comfortably.

Parallelism does not help unless the limit is per-key. Ten workers against a global rate limit is ten workers taking turns, plus a much higher chance of a 429. Check whether the limit is per account, per key, or per endpoint — the answer changes the design and is one line in their documentation.

And the backfill is the binding constraint on every decision. §16.8's re-fetch window looks like a correctness question and is a capacity question: a 30-day window at Kestrel's carrier volume is about 48 requests a day, and a 90-day window is 144. The window you can afford is a function of the rate limit, and the tail has to be handled by a reconciliation rather than by a wider window.

The number worth computing before you write any code: records ÷ page size ÷ rate limit. If the answer is longer than your schedule interval, the design is already wrong and no amount of optimisation inside the client will fix it.

🔐 Privacy & Governance — you are sending data out, not only pulling it in

Every request to a third-party API contains data, and the request is the part nobody classifies.

text what leaves your system in an "ingestion" pipeline ───────────────────────────────────────────────────────────────── the query parameters a tracking number, an order id, an email used as a lookup key the request body on a POST-based search API, a list of customer identifiers the URL path /customers/8841/shipments -- logged by every proxy on the way the User-Agent your service name and version, which is fine and is also fingerprinting

The third row is the one that surprises people. An identifier in a URL path is written to access logs at every hop — your egress proxy, their ingress, any CDN between — and those logs are outside every retention policy either party has agreed. The same identifier in a request body is not.

Three practical rules, in order of how often they are broken:

Prefer a body to a path or query string for anything identifying. It costs nothing and it removes the identifier from four sets of logs you do not control.

Know what the vendor retains. "How long do you keep request logs, and what is in them?" is a question with an answer, usually in a data processing agreement nobody on the engineering team has read. It belongs in the observed contract (Exercise 17.23) alongside the freshness numbers.

And do not log the request yourself. An ingester that logs full request URLs at DEBUG, with DEBUG enabled during an incident, has written a list of customer identifiers into a log aggregator with a different retention and a wider audience than the warehouse. This is the single most common way personal data ends up somewhere nobody catalogued, and the fix is a one-line redaction in the client.

The reciprocal obligation is worth stating too: the data you receive from an API is now yours to classify, retain, and delete (Chapter 31). A carrier's tracking response contains an address. It lands in bronze, it is retained for two years by ADR-003, and nobody decided that — the retention policy was written about clickstream.

🔎 Read the Plan — an API's plan is its response headers

Every other chapter reads an execution plan. An API gives you headers, and they carry more operational information than most clients bother to read.

http HTTP/1.1 200 OK X-RateLimit-Limit: 1000 X-RateLimit-Remaining: 118 X-RateLimit-Reset: 1764547200 Retry-After: 30 Deprecation: Sun, 01 Mar 2026 00:00:00 GMT Sunset: Wed, 01 Jul 2026 00:00:00 GMT Link: <https://api.carrier.example/v3/orders?cursor=eyJ...>; rel="next" ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

Six of those change what your client should do next, and most clients read one.

X-RateLimit-Remaining is the server telling you what your local token bucket cannot know — that another job on your team is consuming the same quota. A limiter that ignores it is correct locally and wrong globally (Exercise 16.12's third test).

Retry-After is not advisory. Retrying before it is exactly the behaviour rate limits exist to stop, and it is what escalates a throttle into a ban (§16.6's case study).

Deprecation and Sunset are a four-month warning that almost nobody logs, and the fix is two lines in the client (§16.9's 🧭).

Link: rel="next" is the pagination contract, and its absence is the termination condition — not an empty page (§16.5).

And ETag is the cheapest incremental mechanism available on an API with no change feed. A conditional request with If-None-Match returns 304 Not Modified with no body, which typically does not count against a quota and always costs less than a full page. On a slowly-changing resource this is the difference between a re-fetch window you can afford and one you cannot.

The instruction: log every header you do not handle, once per header per day. It costs nothing, it is how you discover that the API has been telling you something for six months, and it is the API-shaped version of reading a plan — the system is describing what it did, and the only question is whether anybody is listening.

🎓 Interview Angle — "the API you depend on has no change feed. What do you do?"

A good question, because it has no clean answer and the candidate has to reason about a residual.

The weak answer is "poll it." That is the mechanism, not the design, and it leaves every question unanswered.

The strong answer names three mechanisms and says what each does not cover:

"There are three things you can do and none of them is complete. A re-fetch window over a last-modified parameter catches most changes, and I'd size it from a measurement rather than a guess — store a content hash per record, watch how long after creation records actually change, and pick the window from the distribution. Content hashing tells me what changed once I've re-fetched, so it composes with the window rather than replacing it. And a reconciliation against an aggregate the provider publishes — a daily total, a count — is the only one of the three that can detect a change outside my window. I'd want all three, and I'd write down that the window is a mitigation with a known residual."

Three things that answer does. It names the measurement rather than a number. It says explicitly that hashing does not tell you what to fetch. And it identifies the reconciliation as the only mechanism that covers the tail — which is the insight the question is looking for.

The follow-up is always some version of "how wide is the window?" and the mark is refusing to answer without the measurement, then giving the shape: most changes are in the first day or two, the tail is long and thin, and widening the window to catch the tail is exponentially expensive against a rate limit that does not move (§16.8, and the 📏 above).

A second follow-up worth being ready for: "what if they add a field?" The asymmetry from §16.9 — missing or retyped fields fail; new fields log — and the reason: one breaks you now and the other is information.

And the question behind the question is whether you have been on the wrong end of a rate limit. A candidate who mentions the 429 counter, the Retry-After header, or a ban has operated one of these; a candidate who says "I'd add exponential backoff" has read about one.

🧭 Version Note — what changed in HTTP clients, and what did not

The mechanics in this chapter are stable; the libraries under them are not, and two shifts change the code you should write.

text then now ───────────────────────────────────────────────────────────────────────── requests.Session() httpx, with HTTP/2 and async a manual retry loop tenacity / urllib3 Retry, and the framework's own backoff "just use exponential backoff" full jitter is the default in every serious implementation (§16.5) OAuth2 client credentials still that, plus workload identity federation for machine-to-machine a hand-rolled token cache still hand-rolled, and still where the bugs are (Exercise 16.14)

Two of those rows matter to the design rather than to the imports.

HTTP/2 multiplexing changes the concurrency question. Many requests share one connection, so "open ten connections" is no longer how you go faster — and the rate limit was never about connections anyway (see the 📏 above). The practical effect is that connection pooling stops being a lever and the rate limiter becomes the only one.

And async changes what a rate limiter has to be. A synchronous token bucket protecting a synchronous client is straightforward; the same limiter shared across coroutines needs to be async-aware or it becomes a lock that serialises everything, which is a performance bug that looks like the rate limit working.

What has not changed at all: the four pagination shapes, the retryable/terminal classification, the refresh margin, and the fact that you cannot ask an API what changed. Every one of those is about the protocol and the provider rather than about your client, and a client library upgrade touches none of them.

The reading rule for this subject: library advice ages in about two years and protocol advice does not. A blog post about requests from 2019 is probably obsolete in its imports and probably still correct about Retry-After.

🔁 Idempotency Check — the ingester's own writes, and the API's

Two different idempotency questions live in an API ingester and they are usually conflated.

Your write into bronze must be idempotent, so a retried fetch does not duplicate rows:

```python key = (carrier, tracking_number, response_content_hash)

-> a re-fetch of unchanged data is a no-op; a CHANGED response is a new row

```

The content hash in the key is what makes the re-fetch window (§16.8) affordable. Without it, a 30-day window means re-landing thirty days of unchanged records every night; with it, the write is a no-op for everything that did not change and the window's cost becomes the requests rather than the storage.

And your write to the API — if you make one — must carry an idempotency key, which is a different mechanism entirely and is the API's to honour (§16.6's 🔁).

The test that covers both:

python def test_refetch_is_a_noop(): run_ingest(day="2026-11-27") a = snapshot("bronze.carrier_tracking") run_ingest(day="2026-11-27") # same window, unchanged upstream b = snapshot("bronze.carrier_tracking") assert a == b # both directions, whole rows

Use a recorded fixture for the second run, not the live API — otherwise the test is also a test of whether the carrier changed anything overnight, which it is not for.

16.10 The Kestrel Carrier Ingester

carrier API  (10 req/s, cursor pagination, OAuth2 client credentials)
   │  RateLimiter, steered by X-RateLimit-Remaining
   │  TokenProvider, refreshing 5 min before expiry, thread-safe
   │  retry: classify → backoff with full jitter → honor Retry-After
   ▼
bronze/carrier/v1/fetch_date=YYYY-MM-DD/       raw JSON, unparsed (Ch. 9 §9.4)
   │
   ├── 30-day re-fetch window, nightly     ← catches 99.6% of retroactive changes
   └── monthly reconciliation vs. the carrier's summary  ← catches the rest

🧱 Kestrel Platform — Increment 16: the API ingester

(a) Write platform/ingest/api/carrier.py with the four components: a rate limiter steered by response headers, a token provider with a refresh margin, an error classifier, and cursor pagination that records the last record id as well as the cursor.

(b) Land raw JSON to bronze, unparsed. The envelope from Chapter 9 §9.4.

(c) Prove the retry classifier. Write tests that assert 429 waits for Retry-After, 401 refreshes once and then fails, 400 is never retried, and 503 backs off with jitter. The 400 test is the important one — it is the case that turns one bad request into thousands.

(d) Measure your re-fetch window. Store a content hash per record. After a fortnight, produce the §16.8 table for your own data and choose N from it rather than from this book.

(e) Write the contract test from §16.9 and schedule it nightly.

(f) Add a 429 counter with an alert. This is the control that would have given three days' warning before the ban in §16.6's callout, and it costs one metric.

16.11 Summary

An API is harder than a database in five ways: you cannot see the schema (only what the documentation claims), you cannot see what changed, you are a guest with an enforced quota, every request can fail independently — so retry handling is the main loop, not an edge case — and you cannot test against production.

Four pagination shapes, and three can lose rows. Offset pagination is systematically lossy on a growing collection: inserts shift the collection and records slide across page boundaries, missed entirely, while the duplicates it also produces are invisible because your write is idempotent. Paginate over something that does not move — an ascending sort on an immutable key. Cursor pagination is correct and its cursors expire, so record the last record id too. Page tokens are opaque: never parse one, and terminate on the token's absence rather than on an empty page. Link headers: follow the URL, do not build it.

Read the rate limit from response headers, which are more current than the documentation, and be proactive — a client that runs flat out until throttled spends its life in backoff and may get blocked. Steer from the server's Remaining header rather than only from a local bucket, because other systems share your quota. And compute the wall-clock cost before promising a backfill date: the rate limit, not the data volume, decides whether it is an afternoon or a weekend, and a larger page size or an undocumented bulk endpoint changes it by an order of magnitude.

Tokens expire mid-job. Refresh on a margin — five minutes — not at expiry, and make the refresh thread-safe, because some providers revoke the previous token when a new one is issued, so eight concurrent workers each refreshing produces seven invalid tokens and a cascade that looks like a credential problem.

Backoff needs jitter, and the jitter is not a refinement. Without it, N clients that failed together retry together — backoff merely synchronizes the herd at a longer interval. Full jitter, and always prefer Retry-After to your own calculation.

Which errors you retry matters more than how you back off. 408, 429, 5xx, and network errors are retryable; 400, 401, 403, 404, 409, and 422 are not. Retrying a terminal error is actively harmful: retrying 401 looks like a brute-force attempt, retrying 429 without honoring Retry-After is exactly the behavior rate limits exist to stop, and retrying 400 turns one malformed request into thousands. A blanket retry is a hazard wearing a safe-looking shape, and it got one client blocked for a day — with a 429 counter and an alert, they would have had three days' warning.

Incremental extraction gets one of three things: a modified-since filter (use it, with an overlap window, and assume all four of Chapter 13's lies), an immutable creation ordering (combine with a periodic re-fetch), or nothing (full extraction, or a webhook — and webhooks are lossy in practice, so you need a reconciliation path regardless of what the documentation promises).

Data changes retroactively, with no database analogue and no change signal. Size the re-fetch window by measurement: at Kestrel, 7 days catches 95%, 30 days catches 99.6%, and 90 days costs thirteen times the requests for a further 0.38%. Handle the tail with a different mechanism — a reconciliation against the provider's own aggregate — rather than with a wider window, because widening is expensive and never complete.

Test in three layers. Recorded fixtures are fast and go stale silently. A nightly contract test — one request per endpoint, asserting structure rather than values — is the layer that catches the API changing and the one people skip. And note the asymmetry: a missing or retyped field fails; a new field logs, which is Chapter 13's schema-drift taxonomy applied to an API and is what keeps the check from crying wolf on every provider release.

What's next

Chapter 17 closes Part III with schema evolution and data contracts — the mechanism that turns "they broke us again" from a recurring event into a caught test. It generalizes this chapter's contract test, Chapter 13's schema-drift checker, and Chapter 15's schema registry into one idea: an explicit, versioned, enforced agreement about what a producer will send and what a consumer may rely on.