Case Study 1: The Sawtooth

"Throughput was a third of what one consumer managed alone. We had three consumers. Adding a fourth made it worse."

Executive Summary

Kestrel's bronze clickstream consumer fell behind on a Tuesday afternoon and did not recover. Lag rose to 41 million messages over four hours, oscillating in a sawtooth rather than climbing steadily.

Throughput was about 30% of what a single consumer achieved in isolation, with three consumers running. The team's first response — adding a fourth — made it measurably worse.

The cause was a rebalance storm: a slow downstream write pushed batch processing past max.poll.interval.ms, the coordinator declared a consumer dead, the survivors inherited more work, and they exceeded the interval too.

This case study is §15.5 as an incident. It is here because the symptom does not name the cause, the intuitive response makes it worse, and the fix is a configuration change that takes thirty seconds once you know which one.

Skills applied: rebalancing and the two liveness mechanisms (§15.5); consumer lag as a diagnostic (§15.11); backpressure (Chapter 4 §4.7); the failure whose response amplifies it (Chapter 4 §4.7).

Background

The consumer. bronze-events-writer, three instances, reading kestrel.clickstream.v1 (12 partitions) and writing batches to Delta on object storage.

consumer = Consumer({
    "group.id": "bronze-events-writer",
    "enable.auto.commit": False,
    "max.poll.records": 5000,          # ← tuned up during a throughput push
    "max.poll.interval.ms": 300000,    # 5 min, the default
})

while True:
    msgs = consumer.consume(num_messages=5000, timeout=1.0)
    write_delta_batch(msgs)            # one Delta commit per batch
    consumer.commit(asynchronous=False)

max.poll.records=5000 had been raised from 500 six weeks earlier, during a legitimate throughput improvement. Larger batches meant fewer Delta commits, which meant less transaction-log overhead (Chapter 10 §10.5). Throughput improved 40% and everyone was pleased.

Normal batch time: about 55 seconds. Comfortably inside the five-minute interval, with a margin of roughly 5×.

What changed on the Tuesday. The Delta table's _delta_log had grown — a compaction job had been failing silently for eleven nights (which is its own finding, and is Chapter 10's Case Study 2 in miniature). Snapshot resolution on each write climbed from 2 seconds to about 90.

Batch time went from 55 seconds to roughly 220. Still inside five minutes. Still fine.

Then afternoon traffic arrived.

The Problem

At peak, batches filled to the full 5,000 messages rather than timing out at fewer. Batch time crossed 310 seconds.

14:41  consumer-2 takes 312s on a batch → exceeds max.poll.interval.ms (300s)
14:41  coordinator: "Member consumer-2 has left the group"
       REBALANCE. 12 partitions across 2 consumers instead of 3.
       │
       │  consumer-1 and consumer-3 now have 6 partitions each, not 4.
       │  50% more work per poll cycle.
       ▼
14:47  consumer-1 exceeds the interval. REBALANCE.
       12 partitions on 1 consumer.
       │
14:52  consumer-3 exceeds it. REBALANCE.
       consumer-2 has rejoined by now, so we are back to 3 — and every
       consumer has just discarded its in-flight batch.
       │
14:58  and again.

Four hours of this. Between rebalances, consumption resumed briefly; during them, it stopped entirely. And every rebalance discarded whatever work was in flight, because the offsets had not been committed — so the same messages were fetched and partially processed repeatedly.

⚠️ Failure Mode — The symptom does not name the cause

Consumer lag graphs are the primary Kafka diagnostic, and this incident's graph looked like this:

text lag │ ╱╲ ╱╲ ╱╲ ╱╲ │ ╱ ╲ ╱ ╲ ╱ ╲ ╱ ╲ ← overall trend: rising │ ╱ ╲ ╱ ╲ ╱ ╲ ╱ ╲ │ ╱ ╲╱ ╲ ╱ ╲ ╱ ╲ └──────────────────────────────────────── time

Rising lag and sawtoothing lag mean opposite things and need opposite responses:

Shape Means Response
Steadily rising the consumer is under-provisioned add consumers or partitions
Sawtooth the consumer is rebalancing reduce per-poll work

The team read "rising" and applied the response for rising. Adding a fourth consumer added a rebalance (membership changed) and gave every consumer a smaller share — which sounds like it should help and does not, because the constraint is per-batch time, not aggregate capacity. A smaller share of partitions does not make a 5,000-message batch faster.

The distinguishing check takes one minute:

```bash kafka-consumer-groups --describe --group bronze-events-writer \ | grep -c CONSUMER-ID # how many members RIGHT NOW

run it three times, ten seconds apart. If the count or the assignment

changes, you are rebalancing, not under-provisioned.

```

Or in the consumer logs, which say it plainly and which nobody was reading: Member consumer-2 sending LeaveGroup request... consumer poll timeout has expired

The Analysis

14:41–16:20. The team treated it as a capacity problem. Added a fourth consumer (worse), then a fifth (worse), then began investigating whether object storage was throttling them.

16:20. Someone read the consumer logs rather than the metrics dashboard.

WARN  Member consumer-2-a4f81 sending LeaveGroup request to coordinator
      due to consumer poll timeout has expired. This means the time between
      subsequent calls to poll() was longer than the configured
      max.poll.interval.ms, which typically implies that the poll loop is
      spending too much time processing messages.

The log message names the cause, the setting, and the mechanism. It had been printing every few minutes for ninety-nine minutes.

🔎 Read the Plan — The logs said it in plain English

This is worth sitting with, because it is not a knowledge failure.

The team had a good metrics dashboard, and during an incident they looked at the dashboard. The dashboard showed a number; the log showed a sentence. The sentence named the cause, the configuration setting, and the likely reason, in one paragraph written by someone who anticipated exactly this situation.

The habit worth building: in any incident involving a client library, read the client's own log output before reading your dashboards. Library authors write good diagnostic messages precisely because they know their failure modes, and those messages are frequently more specific than anything you have instrumented.

Kestrel's runbook now opens Kafka incidents with:

```bash

BEFORE the dashboards

kubectl logs -l app=bronze-events-writer --since=15m \ | grep -iE 'rebalanc|leavegroup|poll timeout|coordinator' ```

Which would have ended this incident at 14:45.

16:25. max.poll.records reduced from 5000 to 500. Batches returned to ~30 seconds. Rebalancing stopped immediately. Lag drained over the following 70 minutes.

16:25 onward. The underlying cause — the failed compaction job — was found and fixed the next morning, restoring snapshot resolution to 2 seconds.

The Decision

Five changes.

1. max.poll.records reduced to 500, permanently, with a comment:

# 500, NOT 5000. Larger batches reduce Delta commit overhead and were a real
# 40% throughput win -- and they also push batch time toward
# max.poll.interval.ms, which is a cliff rather than a slope. At 5000 a
# degraded downstream write pushed us past 300s and into a rebalance storm
# that cost four hours. Ch. 15 §15.5.
"max.poll.records": 500,

The comment is the deliverable. Without it, someone will raise it again during the next throughput push, for the same good reason.

2. max.poll.interval.ms raised to 600 seconds. Defense in depth, and the team was explicit that this is treating the symptom: it buys margin and delays detection of a genuinely stuck consumer by five minutes. Accepted deliberately.

3. Static membership. group.instance.id set per instance, so a deploy no longer triggers a rebalance.

This turned out to be the highest-value change and it was added almost as an afterthought. The group had been rebalancing on every deploy — three to five times a week — with a brief consumption pause each time. Nobody had considered that abnormal.

4. Lag shape monitoring, not just lag level. An alert distinguishing a rising trend from an oscillation:

# Rising: the derivative is consistently positive.
# Sawtooth: the derivative alternates sign with high variance.
# They mean opposite things and need opposite responses (§15.5).
recent = lag_samples[-12:]                      # last 12 samples, 10s apart
deltas = [b - a for a, b in zip(recent, recent[1:])]
sign_changes = sum(1 for a, b in zip(deltas, deltas[1:]) if a * b < 0)

if sign_changes >= 4 and max(recent) > LAG_THRESHOLD:
    alert("REBALANCING — lag is oscillating, not rising. Reduce per-poll work; "
          "do NOT add consumers. Check consumer logs for 'poll timeout'.")
elif all(d > 0 for d in deltas):
    alert("UNDER-PROVISIONED — lag rising steadily. Consider more consumers "
          "(up to the partition count) or more partitions.")

The alert text names the response, which is what makes it useful at 04:00.

5. The runbook opens with the consumer logs, before any dashboard.

📐 Design Decision — Batch size is a cliff, not a slope

The team's retrospective identified a mental-model error worth naming.

They had tuned max.poll.records as though it were a slope: bigger batches, better throughput, some diminishing returns, tune to taste. That model is correct right up to max.poll.interval.ms, at which point it is a cliff — and past the cliff, throughput does not degrade, it collapses, and the collapse is self-sustaining.

The general shape: a tuning parameter with a hard timeout downstream of it is not a slope. Others in this book: statement timeout versus query complexity (Chapter 7 §7.3), lock timeouts, HTTP client timeouts versus page size (Chapter 16).

The practical rule they adopted: for any batch-size parameter with a timeout behind it, set the batch so that the p99 processing time is under a third of the timeout. Not half — a third, because the thing that pushes you over is usually a downstream degradation you did not cause and cannot predict, and 5× margin became 1× when snapshot resolution went from 2 seconds to 90.

What Happened

Immediately: lag drained in 70 minutes, throughput returned to normal, and the same three consumers handled peak comfortably at 500 records per batch — about 8% below the peak throughput of the 5,000 configuration, in exchange for never falling off the cliff.

Static membership removed 3–5 rebalances a week that nobody had been counting.

In the eighteen months since:

  • The shape-aware lag alert has fired twice. Once correctly for rebalancing (a consumer with a memory leak slowing down), once correctly for under-provisioning (genuine traffic growth). Both times the alert named the right response.
  • max.poll.records has been questioned twice during throughput work. Both times the comment stopped it.
  • The failed compaction job — the actual trigger — was caught the next morning by the maintenance monitor from Chapter 10, which had been built but not yet alerting. It alerts now.

Lessons

  1. Rising lag and sawtoothing lag mean opposite things and need opposite responses. Read the shape, not just the level.

  2. Adding consumers does not fix per-batch slowness. A smaller share of partitions does not make a 5,000-message batch faster, and the membership change adds a rebalance.

  3. Read the client library's logs before your dashboards. The message named the cause, the setting, and the mechanism, and had been printing for ninety-nine minutes.

  4. A batch-size parameter with a timeout behind it is a cliff, not a slope. Past it, throughput collapses and the collapse is self-sustaining.

  5. Size batches so p99 processing is under a third of the timeout. Margin gets consumed by downstream degradation you did not cause — 5× became 1× when a compaction job failed silently.

  6. Static membership removes deploy-time rebalances, and most teams do not know they are happening.

  7. The comment on the setting is the deliverable. Someone will raise it again for the same good reason, and the comment is what stops them.

  8. An alert should name the response. "Lag is high" is a fact; "reduce per-poll work, do not add consumers" is an action.

  9. The trigger was a different silent failure. A compaction job had been failing for eleven nights. Incidents are usually two problems, and Chapter 4's Case Study 2 said the same thing.

Questions for Discussion

  1. The team looked at dashboards during an incident and the answer was in the logs. Is the fix a runbook item, a log-to-dashboard integration, or training? Argue for one.

  2. max.poll.records=5000 was a legitimate 40% throughput win six weeks earlier. Should the original change have been rejected? What review question would have surfaced the cliff?

  3. The "under a third of the timeout" rule is a heuristic. Where does it come from, and what would make you choose a different fraction?

  4. Static membership removed 3–5 weekly rebalances nobody had counted. How many similar unnoticed-but-costly behaviors do you think a typical streaming pipeline has? How would you find them?

  5. The team accepted that raising max.poll.interval.ms delays detection of a stuck consumer by five minutes. What would make that trade unacceptable?

  6. The shape-aware alert distinguishes rising from oscillating using sign changes in the derivative. What is its false-positive rate on a workload with a strong daily cycle? How would you fix that?

  7. The real trigger was a silently failing compaction job. Trace the dependency: how many systems had to be simultaneously imperfect for this incident to occur? What does that suggest about where to invest?