Exercises: Event Streaming with Apache Kafka

Part D(d) asks you to cause a rebalance storm deliberately. Reading about one and watching the lag sawtooth are different experiences, and the second is what makes §15.5 stick.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

15.1 † State what a Kafka topic is as a data structure, in one sentence. Then derive the four properties in §15.1 from it.

15.2 Give two things Kafka is not, and one thing people commonly try to use it as anyway.

15.3 † What three jobs does the partition key do at once? What does a null key give up?

15.4 Why can partition counts only increase, and what breaks when you increase one?

15.5 † Explain why acks=all without min.insync.replicas can silently degrade to acks=1.

15.6 What is the most common producer bug, and what is its symptom?

15.7 † Why does enable.auto.commit=True silently convert a pipeline to at-most-once?

15.8 Name Kafka's two liveness mechanisms and say which one causes most rebalance problems.

15.9 † Distinguish retention from compaction. Which one makes a topic a table, and why?

15.10 Name the four properties of a dead-letter path that works.

Part B — Standard ⭐⭐

15.11 For each Kestrel topic, choose a partition key and justify it against §15.2's three jobs: kestrel.clickstream.v1, kestrel.orders.cdc.v1, kestrel.inventory.cdc.v1, and a hypothetical kestrel.pricing_changes.v1. For any where the obvious key produces skew, say what you would do.

15.12 † Redo §15.8's sizing for a different consumer: p99 handling time of 25 ms per event, a peak of 900 events/sec, and an expectation of 4× growth in three years. Compute the Little's Law floor, apply the multipliers, then choose a partition count and defend it — including the divisor argument.

15.13 Write the producer from §15.3 with a deliberately broken delivery callback (one that swallows the error). Produce to a topic that does not exist, with auto-creation disabled. Record what you observe, and how long it takes to notice anything is wrong.

15.14 † §15.6 says Kestrel uses cleanup.policy=compact,delete on CDC topics. Explain what each half does, what the combination gives you, and what would go wrong with each alone. Then state the retention value you would choose and why.

15.15 Compute the broker storage for a different topic: 40 million events a day at 1.2 KB, replication factor 3. Produce the retention table from §15.6's 💸 callout for 1, 7, 14, and 30 days. Then answer the question the callout says actually decides it.

15.16 † Take §15.10's dead-letter code and write the replay tool: read the DLQ, group by dlq.reason, print counts, and re-produce a selected subset to the source topic. It must be safe to run twice.

15.17 A consumer group has 12 partitions and 5 consumers. Draw the assignment. Then draw it for 7 consumers, 12 consumers, and 15 consumers. At which point does adding a consumer stop helping, and what does the 5-consumer case tell you about §15.8's divisor argument?

Part C — Deeper ⭐⭐⭐

15.18 §15.8 arrives at 12 partitions when the arithmetic suggested 26, and gives three reasons. Argue that 24 would have been the better choice. What would you need to measure to settle it?

15.19 † Design the monitoring for Kestrel's clickstream pipeline covering §15.11's five signals plus the DLQ. For each: what you measure, the threshold, who is paged, and — the part people skip — what the alert message says, so that the person receiving it at 04:00 knows what to do.

15.20 §15.5's storm is self-sustaining. Model it: with 12 partitions, 3 consumers, a per-event cost that rises with batch size, and max.poll.interval.ms = 300s, at what per-event processing time does the system become unstable? State your assumptions and show the reasoning.

15.21 † The chapter says Kafka transactions give exactly-once "inside Kafka and nowhere else." Research the Kafka Connect sink connectors that claim exactly-once delivery to external systems. What do they actually do, what does it require of the sink, and does it change the claim?

Part D — The Kestrel Platform ⭐⭐⭐

15.22 — Increment 15: the clickstream pipeline.

(a) Add Kafka (or Redpanda) to docker-compose.yml. Create three topics with §15.12's configuration — including the DLQ's 30-day retention, longer than the source's 7.

(b) Write the producer: acks=all, idempotence, zstd, linger.ms=20, and a delivery callback that checks the error. Include a generator replaying Kestrel clickstream events at a configurable rate.

(c) Write the bronze consumer: manual commit after the write, idempotent on (topic, partition, offset), with the §15.10 dead-letter path.

(d) Cause a rebalance storm. Set max.poll.records=5000 and add an artificial per-batch delay that exceeds max.poll.interval.ms. Watch the lag. Then apply each of the three fixes in turn and record which worked and by how much. Also try adding a consumer, and record that it does not help.

(e) Produce a malformed event. Confirm it reaches the DLQ with reason headers intact and that the consumer continued.

(f) Write the replay tool before you need it.

15.23 † Add platform/ingest/stream/lag_monitor.py: reports consumer lag per group per partition, plus the derivative over the last N samples, so a rising lag is distinguished from a sawtoothing one.

That distinction is the whole point: rising lag means a consumer is under-provisioned; sawtoothing lag means it is rebalancing, and the two need opposite responses.

Reflection

A. §15.8 shows the arithmetic giving 26 and the answer being 12, and says so rather than reporting only the formula. When have you seen a calculation presented as an answer when it was really a floor?

B. The DLQ that held 2.1 million messages was "working" by every check anyone had. What else in your systems is technically working and functionally useless?