Chapter 15 — Key Takeaways (Event Streaming with Apache Kafka)

The page for building a consumer, and for the afternoon lag goes sawtooth.

Kafka as a data structure

A topic is an append-only log, partitioned, read by position, and not consumed by reading.

QUEUE                          LOG
[D][C][B][A] ─▶ consumer       [A][B][C][D][E][F][G]
A is GONE. One consumer.         ▲     ▲         ▲
No replay.                    consumer consumer producer
                                 1       2      appends

Four properties follow — the reasons to use it over a queue: replay (reset an offset to fix a consumer bug) · multiple independent consumers · ordering within a partition · durability decoupled from consumption.

Not: a database · a task queue · a system with one global order.

Partitions and keys

Partitions are the unit of parallelism, ordering, AND assignment — three at once, which is why partition count is the most consequential setting.

partition = hash(key) % partition_count

⚠️ Changing the partition count breaks per-key ordering across the change, permanently. Counts can only increase, and increasing one is an ordering decision, not a capacity knob.

Producers

"acks": "all", "enable.idempotence": True,     # + min.insync.replicas=2, RF=3
"compression.type": "zstd", "linger.ms": 20,
producer.produce(..., on_delivery=check_the_error)   # ← NOT optional

⚠️ acks=all with an in-sync set of one is acks=1 wearing a different name. Set min.insync.replicas explicitly.

⚠️ produce() is asynchronous. A producer with no delivery callback silently drops failures — the most common producer bug, and the symptom is missing data with no error.

Consumers

A consumer group assigns each partition to exactly one consumer. So: parallelism is capped by partition count · membership changes trigger rebalances · different groups are independent.

commit BEFORE processing → at-most-once  → a crash LOSES messages
commit AFTER  processing → at-least-once → a crash REPLAYS  ✅

⚠️ enable.auto.commit=True is the DEFAULT and commits on a timer with no knowledge of whether processing succeeded. It silently converts your pipeline to at-most-once.

Rebalancing — the failure you meet first

The storm is self-sustaining: a consumer is declared dead → survivors inherit more work → they exceed the interval too. It does not recover on its own.

Read the SHAPE of the lag graph

Three fixes, in order: 1. Reduce max.poll.records ← do this first, usually sufficient 2. Raise max.poll.interval.ms — treats the symptom, and delays detection of a stuck consumer 3. Process asynchronously with pause()/resume()

Adding consumers does not help — a smaller share of partitions does not make a 5,000-message batch faster.

group.instance.id = static membership → a restart within the session timeout rejoins with its assignment, no rebalance at all. Most teams do not know they rebalance on every deploy.

Read the client library's own logs before your dashboards. "consumer poll timeout has expired... the poll loop is spending too much time processing messages" names the cause, the setting, and the reason.

A batch-size parameter with a timeout behind it is a CLIFF, not a slope. Set the batch so p99 processing is under a third of the timeout — margin gets consumed by downstream degradation you did not cause.

Retention vs. compaction

Compaction is what makes a topic a table. The tombstone (null value) is what removes a key.

Retention is a RECOVERY-WINDOW decision, not a cost decision — how long a consumer can be broken before recovery gets much harder. Same question as max_slot_wal_keep_size (Ch. 14) and vacuum retention (Ch. 10), in three systems.

Kestrel clickstream: 34.4 GB/day at RF=3 → 7 days = 241 GB ≈ $24/month.

Delivery semantics

at-least-once + idempotent writes. Idempotency key: (topic, partition, offset) — unique by construction, needs nothing from the payload.

Kafka transactions give exactly-once inside Kafka and nowhere else.

Sizing

$$L = \lambda W = 2{,}900 \times 0.003 = 8.7 \text{ handlers} \quad\times 2 \text{ growth} \times 1.5 \text{ catch-up} \approx 26$$

Kestrel's topic has 12. The consumer parallelizes internally · 12 has convenient divisors (2,3,4,6) · the peak lasts minutes.

The arithmetic gives you a floor and a shape, not an answer.

Rules: partitions ≥ max consumers in a group · choose many divisors · size for two years · a consumer sized exactly for the arrival rate never catches up · a few hundred partitions per broker is the comfortable ceiling.

Dead letter queues

Four properties: 1. Original bytes, unmodified 2. Reason in headers — dlq.reason, dlq.detail, source topic/partition/offset 3. Alert on RATE, not existence — plus an absolute alert, because a relative one cannot catch a failure present from day one 4. A replay tool, written before you need it

⚠️ Set the DLQ's retention LONGER than the source's. Its contents are what nobody has dealt with yet. The default is shorter, which is backwards — one DLQ held 2.1M messages and had already lost 1.2M to a default nobody set.

"How many are in here" and "how far back does it go" are different questions. If the earliest offset is above zero, you have already lost messages. Applies to every retention-governed store.

Monitoring, in priority order

1. Consumer lag per group per partition — and read its shape
2. Under-replicated partitions
3. Offline partitions
4. Broker disk
5. Request latency p99
6. DLQ rate

An alert should name the response. "Lag is high" is a fact; "reduce per-poll work, do not add consumers" is an action.