Case Study 1: Banned for a Day

"The API told us to stop, five times a request, for four hours. Then it stopped asking."

Executive Summary

Kestrel's carrier ingester retried every non-2xx response five times with exponential backoff. It had run correctly for eight months.

The carrier then introduced a rate limit. The client hit 429, retried without reading Retry-After, hit 429 again, and repeated — five retries per request across a paginated extract. Within four hours the carrier's abuse detection blocked the API key for 24 hours.

Not throttled. Blocked. Every pipeline depending on carrier data stopped, delivery-SLA reporting went dark, and restoring access required a support ticket and an explanation.

This case study is §16.6 as an incident. It is here because the offending code looks entirely reasonable, because the change that triggered it was on the other side of a boundary Kestrel does not control, and because the carrier gave three days of warning that nobody was counting.

Skills applied: error classification (§16.6); Retry-After (§16.3, §16.5); alerting on a rate before it becomes an outage (§16.6); the failure whose response amplifies it (Chapter 4 §4.7).

Background

The client, written in 2024 and reviewed:

def fetch(url, params, attempt=0):
    r = session.get(url, params=params, timeout=30)
    if r.status_code == 200:
        return r.json()
    if attempt < MAX_RETRIES:                    # MAX_RETRIES = 5
        time.sleep(BASE * (2 ** attempt))        # 1, 2, 4, 8, 16 seconds
        return fetch(url, params, attempt + 1)
    raise CarrierError(f"{r.status_code} after {MAX_RETRIES} attempts")

Three things about it are fine. It has a timeout. It backs off exponentially. It gives up eventually rather than looping forever.

Two things are wrong, and both are invisible when the API is behaving:

  • It retries every non-200, with no classification.
  • It ignores Retry-After, using its own exponential instead.

Why it worked for eight months. The carrier had no rate limit. Errors were genuine transient failures — occasional 502s during their deploys — and retrying was exactly right. The client's behavior was correct for the API it was written against.

The Problem

Tuesday, 09:00. The carrier deployed a rate limit: 10 requests per second, returning 429 with Retry-After: 1 when exceeded.

Kestrel's hourly ingester runs at roughly 14 requests per second — it had no limiter, because there had been no limit.

09:00:00  request 1..10   200 OK
09:00:00  request 11      429  Retry-After: 1
          → sleep 1s  (coincidentally correct)
09:00:01  retry           429
          → sleep 2s
09:00:03  retry           429
          → sleep 4s
09:00:07  retry           429
          → sleep 8s
09:00:15  retry           429
          → sleep 16s
09:00:31  retry           429  → give up, raise
          Meanwhile 40 other in-flight requests are doing the same thing.

Each request generated six 429s instead of one. The client was producing roughly six times more rate-limit violations than a client with no backoff at all, because it kept a large number of requests in a retry loop simultaneously.

13:10. The carrier's abuse detection blocked the key.

⚠️ Failure Mode — Backoff without classification amplifies the problem

The intuition is that exponential backoff is a safety mechanism: on error, slow down. That is true only when the error is transient and unrelated to your behavior.

A 429 is neither. It says you are doing too much, and the correct response is to do less — which is what Retry-After specifies. Retrying it on your own schedule, five times, across many concurrent requests, is doing more.

The amplification is arithmetic:

429s generated per logical request
No retry at all 1
Retry 5× ignoring Retry-After 6
Retry honoring Retry-After 2
Proactive limiter, no violation 0

A client with no retry logic at all would have been six times better behaved than this one, and that is the uncomfortable finding: the defensive mechanism made things worse, in the same way backoff without jitter makes a thundering herd worse (Chapter 4 §4.7) and a rebalance makes a slow consumer slower (Chapter 15 §15.5).

The shared shape: a response to failure that increases load. When you add a mechanism that reacts to errors, ask whether its reaction can cause the error it reacts to.

The Analysis

The warning nobody heard. Investigating afterwards, the team found the carrier had introduced the limit with a three-day grace period: from the previous Saturday, 429s were returned but not enforced, and requests succeeded on retry.

Kestrel's client had been receiving 429s for three days. The job succeeded every hour — retries eventually got through — so nothing failed, nothing alerted, and nobody looked.

Sat 09:00   first 429s. Job succeeds after retries. No alert.
Sun         ~4,100 429s. Job succeeds. No alert.
Mon         ~4,300 429s. Job succeeds. No alert.
Tue 09:00   grace period ends. Enforcement begins.
Tue 13:10   BLOCKED.

Roughly 12,000 explicit warnings, ignored, because the only thing being monitored was whether the job succeeded.

🔎 Read the Plan — Count your non-2xx responses, by status

The single control that would have prevented this is one metric:

python metrics.increment("http_response", tags={"host": host, "status": str(r.status_code), "endpoint": endpoint})

And one alert:

yaml - alert: RateLimitedResponses expr: sum(rate(http_response{status="429"}[15m])) by (host) > 0 for: 15m annotations: summary: "Receiving 429s from {{ $labels.host }}. The API is telling us to slow down. Check for a new rate limit, and confirm the client honours Retry-After before this becomes enforcement."

The threshold is zero. Any sustained 429 is a signal, not noise — it is the API explicitly telling you that your behavior is a problem, and there is no rate at which that is acceptable.

This generalizes beyond 429. Systems fire warning shots that pipelines routinely ignore because the job still succeeds:

Warning shot Ignored because
429 retries get through
Deprecation headers (Sunset, Deprecation) nothing breaks yet
PostgreSQL warnings on a query the query returns rows
Kafka log lines about rebalancing (Ch. 15) throughput is merely degraded
Replication lag alerts (Ch. 4 CS 2) they self-resolve

Every one of those has caused an incident in this book, and in every case the signal was present and unwatched. Monitor what the system tells you, not only whether the job exited zero.

The Decision

Five changes.

1. Error classification, before any retry logic.

RETRYABLE = {408, 429, 500, 502, 503, 504}
REFRESH_THEN_FAIL = {401}
TERMINAL = {400, 403, 404, 405, 409, 410, 422}

def classify(status: int) -> str:
    if status in RETRYABLE:
        return "RETRY"
    if status in REFRESH_THEN_FAIL:
        return "REFRESH_ONCE"
    if status in TERMINAL:
        # Retrying these is not merely useless. 401 looks like a brute-force
        # attempt; 429 without Retry-After is exactly what rate limits exist to
        # stop; 400 turns one bad request into thousands. Ch. 16 §16.6.
        return "FAIL"
    return "FAIL" if 400 <= status < 500 else "RETRY"

Note the default. An unknown 4xx fails; an unknown 5xx retries. That asymmetry is deliberate: a 4xx is about your request, and repeating a request the server has rejected is unlikely to help and may be harmful.

2. Retry-After takes precedence, always.

def delay_for(response, attempt: int) -> float:
    ra = response.headers.get("Retry-After")
    if ra is not None:
        # The server knows when it will be ready. Our exponential does not.
        try:
            return float(ra)
        except ValueError:
            return max(0.0, parsedate_to_datetime(ra).timestamp() - time.time())
    return random.uniform(0, min(60.0, 1.0 * (2 ** attempt)))   # full jitter

Retry-After can be seconds or an HTTP date. Both forms are legal and clients that handle only the integer form crash on the other, usually in production, usually during an incident.

3. A proactive rate limiter, steered by response headers (§16.3). The client now runs at 8 requests per second against a 10/s limit and has not received a 429 since.

4. A 429 counter with a zero threshold, and — the broader change — a counter on every non-2xx status by host and endpoint, with a weekly review.

5. A written incident note to the carrier, which mattered more than it should have: the support ticket to restore access went faster because Kestrel could describe exactly what had happened, what they had changed, and when. A vendor deciding whether to unblock you is assessing whether it will happen again.

📐 Design Decision — Retry in the client, or let the job fail and retry the job?

A real alternative the team considered: remove client-level retries entirely. Fail fast on any error, let Airflow retry the whole task.

The case for job-level retry: one retry mechanism instead of two, no possibility of a client amplifying a problem, and the orchestrator already has backoff, alerting, and a limit. It is genuinely simpler.

The case against, which won: a 40,000-request extraction that fails on request 39,000 and restarts from zero is 39,000 wasted requests against a rate limit. At 10 requests/second that is 65 minutes of quota spent on work already done, and with a limited daily quota it can make the extraction impossible to complete at all.

The resolution keeps both, at different granularities: the client retries transient errors on a single request; the orchestrator retries the job on a terminal failure — and the job is resumable (Chapter 13 §13.11), so a restart does not redo the 39,000.

The rule: retry at the smallest granularity that can succeed, and make the next level up resumable. That composition — small retries plus resumable restarts — is what makes both cheap.

What Happened

Access was restored after 22 hours, slightly early, following the support ticket.

The 22-hour gap was backfilled from bronze where possible — carrier data lands raw (Chapter 9 §9.4) — but the missing window had never been fetched, so it required a genuine re-extraction once access returned. A 22-hour gap took 40 minutes to backfill at the new, polite rate.

In the eighteen months since:

  • Zero 429s. The proactive limiter has kept the client under the ceiling continuously.
  • The non-2xx monitor has fired four times. Once for a Deprecation header on an endpoint the carrier retired six months later — caught with five months of notice, which is exactly the warning-shot case that this incident was about missing. Twice for genuine 5xx incidents on the carrier's side. Once for a 403 after a credential rotation.
  • The classifier's FAIL on unknown 4xx has fired once, on a 451, which nobody had enumerated and which the default handled correctly.

The deprecation catch is the outcome the team cites. The incident was about ignoring a signal; the control they built caught a completely different signal five months early.

Lessons

  1. Backoff without classification amplifies the problem. Six 429s per logical request against one for a client with no retry logic at all.

  2. A 429 is not a transient error. It says you are doing too much, and retrying on your own schedule is doing more.

  3. When adding a mechanism that reacts to errors, ask whether its reaction can cause the error it reacts to. Backoff without jitter, rebalancing a slow consumer, and this — the same shape three times.

  4. Retry-After takes precedence over your exponential, and it can be seconds or an HTTP date. Handle both.

  5. Default unknown 4xx to fail and unknown 5xx to retry. A 4xx is about your request.

  6. Count every non-2xx by status, host, and endpoint. The 429 threshold is zero — any sustained one is the API telling you something.

  7. 12,000 explicit warnings were ignored over three days because the only thing monitored was whether the job exited zero. Monitor what the system tells you, not only whether it succeeded.

  8. Retry at the smallest granularity that can succeed, and make the next level up resumable. A job-level retry of a 40,000-request extract wastes 39,000 requests against a quota.

  9. A vendor deciding whether to unblock you is assessing whether it will happen again. Be able to say what changed.

Questions for Discussion

  1. The client's behavior was correct for the API it was written against, and the API changed. Whose responsibility is that? What would a data contract (Chapter 17) change here, if the provider is external?

  2. The 429 alert threshold is zero. Is any non-2xx status a candidate for a zero threshold? Which, and why not the others?

  3. The team found 12,000 ignored 429s. Design the review that would surface signals like this routinely — what it looks at, how often, and who does it.

  4. The 📐 callout's rule is "retry at the smallest granularity that can succeed, and make the next level up resumable." Apply it to three other pipelines in this book. Where does it not hold?

  5. The deprecation header was caught with five months of notice. Estimate how many deprecation and sunset headers your organization currently receives and ignores. How would you find out?

  6. Restoring access took a support ticket and an explanation. What should be in a runbook for "we have been blocked by a vendor"? Who writes the explanation?

  7. The 22-hour gap required real re-extraction because the data had never been fetched. What would a design that survives a vendor outage look like, and is it worth building for a once-in-eighteen-months event?