Chapter 15: Event Streaming with Apache Kafka

"Kafka is a file you can append to and read from many times. Everything else is a consequence."

Overview

Kafka is conceptually simple and operationally demanding, and most confusion about it comes from learning the operational parts before the conceptual one.

The conceptual part fits in a sentence. A Kafka topic is an append-only log, split into partitions, each of which is an ordered sequence of messages that readers consume by position. That is the whole data structure. Producers append; consumers read at their own pace and remember where they got to; nothing is deleted when it is read.

Almost everything else — consumer groups, rebalancing, retention, compaction, delivery semantics — is a consequence of that structure, and understanding it as a consequence is much easier than memorizing it as a feature list.

The chapter spends most of its length on three things that cause most real problems.

Rebalancing (§15.5) is the failure mode you will meet first. A consumer that takes too long between polls is declared dead, partitions are reassigned, consumption stops, and — if the cause was slowness — the reassignment makes it slower, which triggers another rebalance. Chapter 4 §4.7 introduced the sawtooth; here is why it happens and the three settings that fix it.

Sizing (§15.8) is where Chapter 3's Little's Law becomes a partition count. Kestrel's clickstream topic has twelve partitions, and this is the chapter that explains why twelve.

Dead-letter handling (§15.10) is the part most tutorials skip and most production pipelines need within a month, because a message that cannot be processed will arrive and you must decide in advance what happens to it.

In this chapter, you will learn to:

• Describe Kafka as a data structure and derive its properties from it.
• Choose a partition key and predict its ordering and skew.
• Configure a producer for a durability requirement, and price each setting.
• Explain consumer groups and offsets, and why a rebalance stops consumption.
• Diagnose a rebalance storm and name the three settings involved.
• Choose between retention and compaction.
• Size partitions from throughput and service time.
• Design a dead-letter path that neither loses data nor hides problems.

Who needs this chapter: the Streaming and Platform paths, in full. Everyone else should read §15.1, §15.2, and §15.7 — the mental model and the delivery guarantees transfer to every message system you will meet.

15.1 What Kafka Actually Is

A Kafka topic is an append-only log. Producers append to the end. Consumers read from a position they choose and control. Reading does not remove anything.

That last property is the one that distinguishes Kafka from a queue, and it is the source of most of its value:

   A QUEUE                          A LOG (Kafka)
   ┌───┬───┬───┬───┐                ┌───┬───┬───┬───┬───┬───┬───┐
   │ D │ C │ B │ A │──▶ consumer    │ A │ B │ C │ D │ E │ F │ G │
   └───┴───┴───┴───┘                └───┴───┴───┴───┴───┴───┴───┘
   A is consumed and GONE.            ▲       ▲           ▲
   One consumer gets each             │       │           │
   message.                        consumer  consumer   producer
                                      1        2        appends
                                   (offset 0) (offset 3)
   Cannot replay.                  Each reads independently. Both see
   Cannot add a second reader       every message. A new consumer can
   retroactively.                   start from the beginning.

Four properties follow, and they are the reasons to use Kafka rather than a queue:

1. Replay. A bug in a consumer is fixable by resetting its offset and reprocessing. With a queue, the messages are gone. This alone justifies Kafka for most data engineering purposes.

2. Multiple independent consumers. A new consumer added next year can read from the beginning of retention. Adding a consumer costs the producer nothing.

3. Ordering, within a partition. Chapter 4 §4.6's guarantee, and its precise limit.

4. Durability decoupled from consumption. Messages persist for a configured retention period whether or not anyone has read them. A consumer can be down for hours.

What Kafka is not: a database (no queries, no indexes — Chapter 12 §12.9), a task queue (it can be used as one and there are better tools), or a system with a single global order.

15.2 Topics, Partitions, and Offsets

A topic is split into partitions. A partition is an ordered, immutable sequence of messages, each with a monotonically increasing offset.

kestrel.clickstream.v1
  partition 0  [0][1][2][3][4][5]...        ← messages keyed hash(k)%12 == 0
  partition 1  [0][1][2][3]...
  ...
  partition 11 [0][1][2][3][4][5][6]...

Offsets are per partition. Offset 3 in partition 0 and offset 3 in partition 1
are unrelated messages.

Partitions are the unit of parallelism, of ordering, and of assignment. Three things at once, which is why the partition count is the most consequential topic configuration.

The partition key decides everything

A message's key determines its partition: partition = hash(key) % partition_count.

Same key → same partition → ordered relative to each other. Different keys → possibly different partitions → no ordering guarantee between them.

Kestrel's clickstream is keyed by session_id, and that choice does three jobs (Chapter 4 §4.2 established the third):

1. All of one session's events land in one partition, in order — which is what sessionization requires.
2. Different sessions spread across partitions, giving parallelism.
3. session_id has millions of distinct values with no natural hotspot, so the distribution is even.

A null key round-robins, which maximizes even distribution and gives up ordering entirely. Appropriate for metrics; wrong for anything with a per-entity sequence.

Changing the partition count breaks ordering

partition = hash(key) % partition_count. Change the count and the mapping changes for most keys.

Messages already written stay where they are. New messages for a key may land in a different partition. Ordering for that key is broken across the change, permanently, for the messages that straddle it.

Partition counts can only increase, and increasing one is a decision with an ordering consequence, not a capacity knob. §15.8 is about getting it right the first time.

15.3 Producers

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    # Durability. See the table below -- this is the setting that decides
    # whether an acknowledged write can be lost.
    "acks": "all",
    "enable.idempotence": True,      # exactly-once *to Kafka*. Ch. 4 §4.5.
    "compression.type": "zstd",
    "linger.ms": 20,                 # batch for 20ms before sending
    "batch.size": 262144,
    "max.in.flight.requests.per.connection": 5,   # safe with idempotence on
})

def on_delivery(err, msg):
    """Called asynchronously. NOT calling this -- or not checking err -- is the
    most common producer bug: produce() returns immediately and success is
    reported later, so a producer with no delivery callback silently drops
    failures."""
    if err is not None:
        metrics.increment("produce_failed", tags={"topic": msg.topic()})
        log.error("delivery failed", topic=msg.topic(), error=str(err))
        dead_letter(msg, err)

producer.produce(topic="kestrel.clickstream.v1",
                 key=event["session_id"].encode(),
                 value=serialize(event),
                 on_delivery=on_delivery)
producer.poll(0)        # serve delivery callbacks

acks, and what each level costs

acks=all with min.insync.replicas=2 and replication.factor=3 is the durable configuration, and it is what Kestrel uses for anything that feeds a revenue number.

min.insync.replicas is the setting people omit. With acks=all and no minimum, a partition whose replicas have all fallen out of sync has an in-sync set of one — and acks=all then means "acks from the single remaining replica," which is acks=1 wearing a different name. Set the minimum explicitly.

The three producer settings that surprise people

enable.idempotence=True gives exactly-once delivery to Kafka — the producer attaches a sequence number and the broker deduplicates retries. It says nothing about your consumer or your sink (Chapter 4 §4.5). It is nearly free and should be on.

linger.ms trades latency for throughput by batching. The default of 0 sends immediately, which produces many small requests. 20 ms is usually a large throughput win for a latency cost nobody perceives.

produce() is asynchronous. It returns before the message is sent. A producer that does not poll for delivery callbacks and check the error silently drops failures, and the symptom is missing data with no error anywhere — the most common producer bug and the reason the callback above is not optional.

15.4 Consumers and Consumer Groups

consumer = Consumer({
    "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP"],
    "group.id": "bronze-events-writer",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,        # commit AFTER writing. See below.
    "max.poll.interval.ms": 300000,     # 5 min. §15.5.
    "max.poll.records": 500,
})
consumer.subscribe(["kestrel.clickstream.v1"])

while True:
    msgs = consumer.consume(num_messages=500, timeout=1.0)
    if not msgs:
        continue
    write_to_bronze(msgs)      # idempotent
    consumer.commit(asynchronous=False)    # commit only after the write lands

A consumer group is a set of consumers sharing a group.id. Kafka assigns each partition to exactly one consumer in the group.

Three consequences:

Parallelism is capped by partition count. Twelve partitions means at most twelve useful consumers in a group. A thirteenth sits idle.

Adding or removing a consumer triggers a rebalance. §15.5.

Different groups are independent. Two groups both read every message and track offsets separately, which is how a bronze writer and a real-time alerting service consume the same topic without interfering.

Offset commits — the ordering that decides your delivery semantics

Commit after processing, not before. The code above does, and the alternative — enable.auto.commit=True — commits on a timer regardless of whether processing succeeded.

  commit BEFORE processing  →  at-most-once  →  a crash LOSES messages
  commit AFTER  processing  →  at-least-once →  a crash REPLAYS messages

At-least-once plus an idempotent write is the pattern (Chapter 4 §4.5), which means the second ordering, always, for anything that matters.

enable.auto.commit=True is the default, and it commits every auto.commit.interval.ms from a background thread with no knowledge of whether your processing succeeded. It is the setting that silently converts your pipeline to at-most-once.

15.5 Rebalancing

The failure mode you will meet first, and the one whose symptom is least self-explanatory.

What happens

When group membership changes, the group coordinator reassigns partitions. During a rebalance, consumption stops for the whole group.

Membership changes when a consumer joins, leaves, or is declared dead — and the third is where the trouble is.

The two heartbeats

Kafka uses two independent liveness mechanisms and confusing them is why rebalance problems are hard to diagnose:

The second is the one that bites. Your process is alive and heartbeating fine. It is simply taking longer than five minutes to process a batch — a slow database write, a large batch, a downstream service degrading — and the coordinator concludes it has stalled.

The storm

  consumer 3 takes 6 min on a batch (a slow warehouse write)
      │
      ▼  exceeds max.poll.interval.ms → declared dead
  REBALANCE: 12 partitions redistributed over 2 consumers instead of 3
      │
      ▼  the remaining two now have 50% more work each
  they now exceed max.poll.interval.ms too
      │
      ▼  REBALANCE. And again.
  Throughput collapses to near zero while every consumer is repeatedly
  declared dead and reassigned.

This is Chapter 4 §4.7's sawtooth, and its defining property is that the system's response to slowness makes it slower. It does not recover on its own.

⚠️ Failure Mode — Diagnosing a rebalance storm

The symptoms, and none of them says "rebalance":

• Consumer lag rises and falls in a sawtooth rather than trending.
• Throughput far below what a single consumer achieves alone.
• Log lines about group coordination, joining, and leaving.
• Records processed more than once — because the offsets of an in-flight batch were never committed.

The three settings, and the order to apply them:

1. Reduce max.poll.records. The fastest fix and usually sufficient. 500 → 100 makes each poll's work five times smaller, so it comfortably fits the interval. Do this first.
2. Raise max.poll.interval.ms to comfortably exceed your worst-case batch time. This is treating the symptom, and it is legitimate when your processing genuinely is slow and bounded. Raising it also delays detection of a genuinely stuck consumer, which is the cost.
3. Process asynchronously — poll, hand off to a worker pool, poll again — and use pause()/resume() for backpressure. Correct, more complex, and it is what you do when 1 and 2 are not enough.

What people try that does not work: adding consumers. If the cause is per-batch slowness, more consumers each still take too long, and you have added rebalance churn.

And the setting worth knowing about: group.instance.id enables static membership, so a consumer restarting within session.timeout.ms rejoins with its existing assignment and no rebalance happens at all. For a consumer group that is restarted on every deploy, this removes a whole class of churn for one line of configuration.

15.6 Retention and Compaction

Two policies, and they answer different questions.

Retention — cleanup.policy=delete — keeps messages for a time or size limit, then deletes the oldest. "Keep the last seven days." This is the default and it suits event streams.

Compaction — cleanup.policy=compact — keeps the most recent message per key, forever. "Keep the current state of every entity." This suits changelogs and CDC.

  RETENTION (7 days)              COMPACTION
  k=A v=1  ← aged out             k=A v=1  ← superseded, removed
  k=B v=1  ← aged out             k=B v=1  ← superseded, removed
  k=A v=2                         k=A v=2  ← superseded, removed
  k=A v=3  ← kept                 k=A v=3  ← KEPT (latest for A)
  k=B v=2  ← kept                 k=B v=2  ← KEPT (latest for B)

Compaction is what makes a topic a table. A compacted CDC topic, replayed from the beginning, reconstructs the current state of every row — which is why Chapter 14's kestrel.orders.cdc.v1 is compacted.

The tombstone matters here. A message with a null value tells compaction to remove the key entirely (Chapter 14 §14.7). Without tombstones, a compacted topic retains deleted keys forever.

Kestrel uses both on one topic: cleanup.policy=compact,delete on the CDC topics, with a 7-day delete window. Latest state per key is retained; individual old versions age out. That combination is not obvious and it is what you usually want for CDC.

💸 Cost Check — Retention is a storage decision with a replay consequence

Kestrel's kestrel.clickstream.v1: 14M events/day at ~820 bytes, replication factor 3.

$$14{,}000{,}000 \times 820 \times 3 = 34.4 \text{ GB/day of broker storage}$$

Retention	Broker storage	What it buys
1 day	34 GB	recover from a consumer outage of hours
7 days	241 GB	recover from a weekend, reprocess a week
30 days	1.03 TB	reprocess a month without touching bronze

At managed-Kafka storage of $0.10/GB-month, 7 days is about $24/month and 30 days about $103/month.

Retention is not primarily a cost decision — it is a recovery-window decision. It bounds how long a consumer can be broken before you must reprocess from bronze instead, and it bounds how far back you can reset an offset to fix a bug.

Kestrel's 7 days is chosen to cover a long weekend, which is the same reasoning as max_slot_wal_keep_size in Chapter 14 §14.6 and the same reasoning as vacuum retention in Chapter 10 §10.5. Three different systems, one question: how long can this be broken before the recovery gets much harder?

15.7 Delivery Semantics in Practice

Chapter 4 §4.5's material, made concrete.

Kafka's transactional producer genuinely provides exactly-once for consume-process-produce inside Kafka. The moment your consumer writes to PostgreSQL, S3, or a warehouse, you are outside the transaction boundary and it is at-least-once — Chapter 4 §4.5's diagram, unchanged.

So: at-least-once plus idempotent writes. For Kestrel's bronze writer, the idempotency key is (topic, partition, offset), which is unique by construction and requires no cooperation from the event payload.

🔁 Idempotency Check — (topic, partition, offset) is unique, and that is not the same as safe

The key is unique by construction, and two things can still go wrong with it.

It is not stable across a topic migration. Re-key the topic, change the partition count, or replay into a new topic name, and every offset changes. The same event now carries a different idempotency key, and a writer that dedupes on it will happily write the event a second time. Kestrel hit this during the 8 → 12 partition change (§15.8) and the bronze table gained 1.4 million duplicate rows in an afternoon.

And it is not the event's identity. It identifies this delivery of this event on this topic. If the producer sends the same logical event twice — a retried HTTP call upstream, a mobile client resending on reconnect — the two deliveries have different offsets and both are written.

The distinction is worth stating precisely, because both are called deduplication:

text (topic, partition, offset) -> protects against REDELIVERY consumer-side a producer-assigned event_id -> protects against RE-SENDING producer-side

You need both, and only one of them is free. Kestrel's clickstream events carry a client- generated event_id (a UUID, minted once at the moment of the interaction and reused on every retry), and bronze dedupes on event_id while the writer dedupes on the offset triple.

Test it the way Chapter 4 §4.6 says to: run the consumer twice over the same offsets and diff the table. clickstream_consumer.py --self-check does exactly this, and it is nine lines of test for a class of bug that otherwise surfaces as a revenue figure being 0.3% high.

15.8 Sizing Partitions

Where Chapter 3 §3.5's Little's Law becomes a number.

The calculation

Kestrel's clickstream: 2,900 events/sec at peak. The bronze consumer's p99 handling time, measured, is 300 ms per batch of 100 — so 3 ms per event at the tail.

$$L = \lambda W = 2{,}900 \times 0.003 = 8.7 \text{ concurrent handlers}$$

That is the minimum. Three multipliers apply, and each is a judgment:

Headroom for growth. Partition counts can only increase, and increasing breaks per-key ordering (§15.2). Size for where you will be in two years, not today. Kestrel assumed 2×.

Headroom for catch-up. After an outage, the consumer must process faster than real time to catch up. A consumer sized exactly for the arrival rate never catches up. Kestrel assumed 1.5×.

Rebalance granularity. With 8 partitions and 3 consumers, the assignment is 3-3-2 — one consumer does 50% more work than another. More partitions gives finer distribution.

$$8.7 \times 2 \times 1.5 \approx 26 \to \text{round to } 12 \text{?}$$

No — and this is where the chapter has to be honest. Kestrel's topic has 12 partitions, which is below what that arithmetic suggests. The reasoning recorded in the ADR:

• The 3 ms p99 was measured on a batch write to object storage, which parallelizes within a consumer. One consumer with 4 worker threads handles ~1,300 events/sec, so 12 partitions across 3 consumers is ample.
• 12 has convenient divisors — 2, 3, 4, 6 — so consumer counts distribute evenly. 26 divides badly.
• The peak is 2,900/sec for minutes, not hours, and a few minutes of lag at peak is acceptable under Kestrel's SLA.

The arithmetic gives you a floor and a shape, not an answer. Reporting only the formula would have been tidier and would have taught the wrong thing.

The rules that generalize

• Partitions ≥ maximum expected consumers in any one group.
• Choose a number with many divisors — 12, 24, 60.
• More partitions is not free: more open file handles, more memory per broker, longer leader elections, and longer rebalances.
• A few hundred partitions per broker is a comfortable ceiling on modern Kafka; thousands is a known pain point.

📏 Scale Note — where each of these numbers stops working

Every rule in this section has a scale at which it reverses. Kestrel is a mid-sized platform; here is the map, so you know which advice you have outgrown.

	Kestrel	Where it changes	What changes
Events/sec	2,900 peak	~100k/sec	one topic is no longer one team's problem; you need quotas
Partitions/topic	12	~200	rebalance time becomes the operational constraint
Partitions/broker	~90	a few thousand	leader elections and recovery get slow and scary
Topics	~40	~1,000	naming, ownership, and retention need policy, not convention
Retention	7 days	months	you are running a storage system; consider tiered storage
Consumer groups	9	~100	the coordinator becomes a hotspot; group ids need a scheme

The row that surprises people is retention. At seven days Kafka is a buffer and the broker disk is an implementation detail. At ninety days it is a storage system with a replication factor of three — 3× the bytes on the most expensive storage in the platform, which is §15.6's Cost Check at a different scale entirely.

And notice what is not on this list: throughput per broker. Kafka is very rarely throughput-bound before it is partition-count-bound or retention-bound. The constraint that bites is almost always metadata, not bandwidth, and sizing conversations that start with MB/sec are usually answering the wrong question.

15.9 Schemas and the Registry

A Kafka message is bytes. Something must agree on what they mean.

A schema registry stores schemas by id; producers write the id in the message and consumers look it up. The consequence that matters, from Chapter 11 §11.5:

With a registry, an incompatible change fails at the producer, at write time, before the bad data exists. Without one, it fails in the consumer, months later.

The registry enforces a compatibility mode — backward, forward, or full — and rejects a schema that violates it. Chapter 17 covers this properly; the point here is that it is the only mechanism in this chapter that moves a failure from the consumer to the producer.

15.10 Dead Letter Queues

A message will arrive that your consumer cannot process. Malformed, a schema you cannot resolve, a value that breaks a downstream constraint. You must decide in advance what happens, and there are only three options:

Dead-lettering is right, and only if someone reads the dead-letter topic.

def handle(msg) -> None:
    try:
        event = deserialize(msg.value())
        validate(event)
        write_to_bronze(event)
    except (DeserializationError, ValidationError) as exc:
        producer.produce(
            topic="kestrel.clickstream.dlq.v1",
            key=msg.key(),
            value=msg.value(),                 # the ORIGINAL bytes, unmodified
            headers={
                "dlq.reason": type(exc).__name__,
                "dlq.detail": str(exc)[:500],
                "dlq.source_topic": msg.topic(),
                "dlq.source_partition": str(msg.partition()),
                "dlq.source_offset": str(msg.offset()),
                "dlq.failed_at": now_iso(),
                "dlq.consumer_version": VERSION,
            })
        metrics.increment("dlq", tags={"reason": type(exc).__name__})

Four properties of a dead-letter path that works:

1. The original bytes, unmodified. You will want to reprocess after fixing the consumer, and a re-serialized message is not the message that failed.

2. The reason, in headers. Why it failed, and enough context to find it. Without this the DLQ is a bucket of bytes nobody can triage.

3. An alert on DLQ rate, not on DLQ existence. A steady trickle is normal; a spike is an upstream change.

4. A replay tool, written before you need it. Reading the DLQ, fixing, and re-producing to the main topic is a routine operation and it should not be improvised during an incident.

⚠️ Failure Mode — The dead-letter topic nobody read

A team implemented dead-lettering correctly: original bytes, reason headers, a separate topic. They did not implement the alert or the replay tool.

Fourteen months later the DLQ held 2.1 million messages. A schema change eleven months earlier had made a subset of events unparseable, and every one of them had been quietly diverted.

Three things this cost:

• The data was not lost, which is the DLQ working. It was also not available, which is the DLQ failing.
• Kafka retention had aged out the oldest 8 months. The DLQ had the default 7-day retention because nobody had configured it — so the messages that had been "safely diverted" were gone.
• Nobody could reprocess the remainder, because no replay tool existed and writing one against a schema that had since changed twice took four days.

A dead-letter queue with no alert and no replay tool is a slower way to lose data. Three configuration lines and a fifty-line script, written when you build the DLQ, are the difference. And set the DLQ's retention longer than the main topic's, not shorter — its contents are, by definition, the things you have not dealt with yet.

15.11 Operating Kafka

Briefly, because Kestrel uses managed Kafka and so should most four-person teams (Chapter 5 §5.3).

What you monitor, in priority order:

1. Consumer lag, per group per partition. The single most important Kafka metric. Rising lag means a consumer is falling behind; sawtooth lag means rebalancing (§15.5).
2. Under-replicated partitions. Any non-zero value means a broker is struggling or down.
3. Offline partitions. Non-zero means unavailability.
4. Broker disk. Retention is a promise you make about disk you have.
5. Request latency, producer and consumer, p99.

What you can safely not do, if you use a managed service: broker sizing, rack awareness, ZooKeeper or KRaft operation, partition reassignment, and rolling upgrades. This is most of the operational burden and it is the strongest argument for managed Kafka at small team sizes.

🧭 Version Note — ZooKeeper is gone, and half the internet has not noticed

Kafka's metadata layer was replaced. KRaft — Kafka's own Raft-based controller quorum — became production-ready in 3.3, the default for new clusters in 3.5, and ZooKeeper support was removed entirely in 4.0. There is no ZooKeeper in a current Kafka cluster.

Why it matters to a reader rather than an operator: an enormous amount of the Kafka material you will find — blog posts, Stack Overflow answers, tuning guides, and a good deal of the conference talk canon — assumes a ZooKeeper ensemble, and its advice about controller failover, metadata propagation, and cluster sizing describes a system that no longer exists.

text a page that mentions is describing ───────────────────────────────────────────────────────────────── zookeeper.connect Kafka 3.4 or earlier --zookeeper on a CLI pre-2.2 habits, or copied from them a 3- or 5-node ZK ensemble an architecture you will not deploy KRaft, controller.quorum.* current

The practical test when you find advice online: search the page for "zookeeper." If it is there and undated, treat everything operational on the page as suspect and everything conceptual on it as probably still fine. Partitions, offsets, consumer groups, and rebalancing are unchanged — which is why this chapter spends its pages there.

🧪 Try It — cause a rebalance storm on purpose

bash cd part-03-ingestion/chapter-15-event-streaming-with-kafka/code python clickstream_consumer.py --self-check python clickstream_consumer.py --break-it

--break-it sets max.poll.interval.ms below the handler's actual processing time, which is §15.5's storm, reproduced in one flag. Watch the sequence in the log:

text poll -> handle 100 events -> exceed max.poll.interval.ms -> the coordinator evicts this consumer -> rebalance; partitions reassigned -> the same batch is delivered to a different consumer -> which also exceeds it

Nothing is failing. No exception, no error log from the handler, no dead broker. The consumer group simply never makes progress, and lag climbs on a topic whose producers are perfectly healthy — which is why §15.5 insists that "rebalancing constantly" and "broken" look identical from outside.

Then fix it two ways and compare. Raise max.poll.interval.ms, and separately lower max.poll.records. Both stop the storm; only one of them is a diagnosis, and deciding which is the exercise.

🎓 Interview Angle — "how do you guarantee exactly-once with Kafka?"

This is a trap question and it is asked sincerely, usually by someone who has read the transactional-producer documentation and not the fine print.

The weak answer is "enable transactions and enable.idempotence." That is exactly-once within Kafka: consume-process-produce, Kafka to Kafka. The moment the consumer writes to S3, a warehouse, or Postgres, the write is outside the transaction and you are back to at-least-once (§15.7, and Chapter 4 §4.5).

The strong answer refuses the premise and then answers the real question:

"Inside Kafka, yes — transactions plus idempotent producers give you exactly-once for a consume-process-produce loop. But our sink is object storage, so the write isn't in that transaction. What I'd actually build is at-least-once delivery plus an idempotent write: dedupe on a producer-assigned event id, key the write on something stable, and make replaying the same offsets produce the same table. Then I'd test that by running the consumer twice over the same range and diffing."

The follow-up that separates people: "what's your idempotency key?" A candidate who says (topic, partition, offset) and stops has not thought about a partition-count change. A candidate who names both keys and says what each protects against has run this in production.

🏭 From the Pipeline — the topic that was fine until somebody added a consumer

A clickstream topic had one consumer: the bronze writer. It ran for eighteen months without incident.

A second team added a consumer — a real-time dashboard, in its own consumer group, reading the same topic. Correct, by design; a topic is meant to support this (§15.1's fourth property).

Within a day, the bronze writer's lag was climbing and the brokers' disks were filling.

What happened is that nothing about the new consumer was the problem. The new consumer was fine. What changed was the read pattern on the brokers: the bronze writer read from the tail, always, which meant its reads were served from the page cache. The dashboard consumer, on being deployed, started from earliest — seven days of history — and pulled the entire retention window through the page cache, evicting the tail.

The bronze writer's reads started hitting disk. Its throughput dropped by roughly 60%, lag climbed, and because lag climbed the writer fell further from the tail, which made its reads cold too.

text the feedback loop, which is the part worth understanding a cold consumer evicts the cache -> the warm consumer's reads go to disk -> the warm consumer slows -> the warm consumer falls behind the tail -> ITS reads are now cold too

The disk filling was a second-order effect: the brokers were compacting and flushing more, and the retention could not be enforced as fast as data arrived.

It resolved on its own in about six hours, when the dashboard consumer caught up to the tail — which is why it was nearly diagnosed as a transient network problem.

Three lessons, and the third is the general one.

Bootstrapping a consumer from earliest on a busy topic is a capacity event. It is a full-retention read at maximum rate, and it should be scheduled rather than deployed on a Tuesday afternoon. Most clients support a rate limit; almost nobody sets one.

Consumers on a shared topic are not isolated. They share brokers, disks, and page cache. The "multiple independent consumers cost nothing extra" property (§15.1) is true about offsets and false about I/O, and the distinction is only visible under load.

And a system with a positive feedback loop has no small version of its failure. Lag that causes more lag does not degrade gracefully; it goes from fine to an incident in minutes and then recovers abruptly, which is exactly the shape that gets misdiagnosed as external.

📐 Design Decision — one topic per event type, or one topic for everything?

Kestrel's clickstream is one topic carrying nine event types. The alternative — nine topics — is defensible and is what a lot of teams do, and the decision is harder than it looks because the two options fail differently.

text one topic, many types one topic per type ───────────────────────────────────────────────────────────────────────── ordering across types GUARANTEED per key none. A purchase can be consumed before the page view that preceded it. a consumer wanting one type filters; reads everything subscribes; reads only what it needs adding a type no operational change a new topic, ACLs, monitoring, retention schema a union, or an envelope one schema per topic; with a payload the registry is happier partition count sized for the total sized nine times, and eight of them wrong retention one policy per type, which is sometimes what you want

The ordering row is the one that decides it here. Sessionisation (Chapter 18 §18.9) requires events for one anonymous_id in order across types — a page view, then an add-to-cart, then a purchase. Split across nine topics, that ordering does not exist, and reconstructing it means buffering and sorting by a client timestamp you do not trust (Chapter 29).

The cost of the choice is the filtering row, and it is real: a consumer interested only in purchases reads 14 million events a day to find perhaps 6,600. At Kestrel that is cheap. At ten times the volume it is a reason to publish a derived, filtered topic — which is the resolution, and it is a projection (Chapter 36) rather than a re-partitioning of the source.

The rule that generalises: partition topics by what must be ordered together, not by what is consumed together. Ordering is a property you cannot add later; filtering is one you can.

And the honest counter-case: if the types have genuinely different retention requirements, or different access-control requirements, those are not solvable with a filter and they are a legitimate reason to split. Kestrel's nine types have neither.

🔐 Privacy & Governance — a log you cannot delete from

A Kafka topic is append-only by design, and every property that makes it useful — replay, retention, multiple independent consumers — makes selective deletion impossible.

There is no DELETE FROM topic WHERE customer_id = 8841. There are exactly three mechanisms and each has a cost:

text mechanism what it does the cost ──────────────────────────────────────────────────────────────────────── retention deletes by AGE, everything you wait; and the older than the window window bounds replay compaction + a tombstone deletes prior versions of ONE requires a key, and it key, once compaction runs is EVENTUAL crypto-shredding encrypt per subject, then key management, and destroy the key every consumer must decrypt

Retention is the mechanism you will actually use, and it means an erasure request against a topic is satisfied by waiting, not by acting. A 7-day retention makes that acceptable. A 90-day one makes it a conversation.

Compaction plus a tombstone is the only selective mechanism, and it has two traps. It needs the topic to be keyed by the subject — a clickstream keyed by anonymous_id can tombstone a browser and not a person — and compaction is eventual, so the record survives in the uncompacted tail for an unbounded time.

Crypto-shredding works and is the only one that does. Encrypt each subject's payload with a per-subject key held outside the log; destroying the key makes every copy — in the topic, in the replicas, in any consumer's replay, in a backup — permanently unreadable at once. It is also a real key-management system that every consumer depends on, which is why it is rare.

Two design decisions in this chapter that are privacy decisions:

The retention (§15.6). Chosen for replay and bootstrap; it is also the deletion latency.

And the partition key (§15.3). Chosen for ordering; it is also the only granularity at which compaction can ever remove anything. A null key gives up ordering, compaction, and any possibility of selective deletion — three things, one omission.

The practical instruction: write the topic's retention and key into the contract (§17.4) with the privacy consequence stated. "7 days; a deletion request is satisfied within 7 days by expiry" is a sentence somebody in a compliance conversation can use, and it takes ten seconds to write now.

🔎 Read the Plan — four commands that answer "what is this topic doing"

```bash

1. the topic's configuration -- the numbers that decide everything

kafka-topics.sh --describe --topic kestrel.clickstream.v1 --bootstrap-server $B

PartitionCount: 12 ReplicationFactor: 3

Configs: cleanup.policy=delete,retention.ms=604800000,min.insync.replicas=2

2. lag, per partition -- the single most important metric (§15.11)

kafka-consumer-groups.sh --describe --group bronze-writer --bootstrap-server $B

PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID

0 41208113 41208140 27 consumer-1-...

3 38102441 39104882 1002441 consumer-4-... <- !

3. is the data where you think? read one message and look at it

kafka-console-consumer.sh --topic kestrel.clickstream.v1 \ --partition 3 --offset 38102441 --max-messages 1 \ --property print.headers=true --bootstrap-server $B

4. partition SIZES -- skew in the topic, not in the consumer

kafka-log-dirs.sh --describe --bootstrap-server $B \ --topic-list kestrel.clickstream.v1 | jq '.brokers[].logDirs[].partitions[]' ```

Command 2 is the one to run first and the one people misread. Lag on one partition and not the others is not a throughput problem — it is a skewed key, a stuck consumer, or a poison message on that partition, and adding consumers cannot help because a partition is consumed by exactly one.

Command 4 answers the question command 2 raises. Wildly uneven partition sizes mean the key is skewed (§15.3); even sizes with uneven lag mean the consumer on that partition is the problem.

And command 3 is the one nobody runs, because it feels intrusive. Reading the message at the offset where the consumer is stuck takes ten seconds and frequently ends the investigation — a malformed payload, a schema id that is not in the registry, a field that is null and should not be.

The general instruction: a topic will tell you what it holds. The reluctance to look at production data is a good instinct and the wrong one during an incident, and the fix is to look at one message rather than to avoid looking.

🎓 Interview Angle — "why did you choose Kafka?"

A question about judgment, and the trap is that Kafka is a good answer to a question that may not have been asked.

The weak answer lists features: durable, ordered, replayable, scalable. All true, and none of it explains why this problem needed them.

The strong answer names the property that forced it:

"We needed multiple independent consumers over the same events, and we needed to be able to replay — a new consumer bootstrapping from the beginning, or reprocessing after a bug. A queue gives you neither: reading is destructive and there is one consumer group's worth of state. That's really the whole reason. If we'd had one consumer and no replay requirement, a queue or even a table with a watermark would have been simpler and I'd have argued for it."

The concession at the end is the mark. A candidate who can say when Kafka would have been the wrong choice has evaluated it; one who cannot has adopted it.

The follow-ups, and what each is checking:

"What ordering guarantee do you have?" — per partition, per key. A candidate who says "the topic is ordered" has not operated one.

"What happens when a consumer falls behind?" — lag, then the retention window, then data loss. The good answer mentions that lag and rebalancing look identical from outside and need opposite responses (§15.5).

"How do you avoid duplicates?" — you do not; you make the write idempotent. This is Chapter 4 §4.7 and it is the single most reliable discriminator in a streaming interview, because the textbook answer is "enable exactly-once" and the practitioner answer is "at-least-once plus a keyed write."

And "how would you size the partitions?" — Little's Law, plus growth, plus catch-up, rounded to a number with many divisors, and the observation that you can only ever increase it and increasing breaks per-key ordering (§15.8). That last clause is what separates a computed answer from a memorised one.

15.12 The Kestrel Clickstream Pipeline

web / iOS / Android
   │  producer: acks=all, idempotence, zstd, linger 20ms
   ▼
kestrel.clickstream.v1        12 partitions · key=session_id
                              replication 3 · min.insync.replicas 2
                              retention 7 days · cleanup.policy=delete
   │
   ├──▶ group: bronze-events-writer   ──▶ bronze/events/ (Delta, Ch. 10)
   │       commit after write · idempotency key (topic, partition, offset)
   │
   ├──▶ group: realtime-inventory     ──▶ stock reservation (Ch. 29)
   │
   └──▶ (failures) ──▶ kestrel.clickstream.dlq.v1
                       3 partitions · retention 30 days ← LONGER than the source

Three decisions in that diagram are worth defending:

Two consumer groups on one topic. The bronze writer and the real-time inventory service read the same events independently, at different speeds, with separate offsets. Neither can affect the other. This is property 2 from §15.1 and it is why the topic is not a queue.

The DLQ's retention is 30 days against the main topic's 7. Its contents are the things nobody has dealt with yet, so it needs more time, not less. The reverse — which is the default — is the mistake in §15.10's callout.

session_id as the key, not event_id or null. Ordering within a session is what sessionization needs; event_id would give perfect distribution and no useful ordering.

🧱 Kestrel Platform — Increment 15: the clickstream pipeline

(a) Add Kafka (or Redpanda) to docker-compose.yml. Create the three topics with the configuration above — including the DLQ's longer retention.

(b) Write the producer: acks=all, idempotence, a delivery callback that checks the error, and a generator that replays Kestrel clickstream events at a configurable rate.

(c) Write the bronze consumer: manual commit after the write, idempotent on (topic, partition, offset), with the dead-letter path from §15.10.

(d) Cause a rebalance storm. Set max.poll.records=5000 and add an artificial delay so a batch exceeds max.poll.interval.ms. Watch the lag sawtooth. Then fix it with each of the three settings in turn and record which worked and by how much. This is the exercise — reading about a rebalance storm and seeing one are different.

(e) Produce a malformed event. Confirm it lands in the DLQ with its reason headers intact, and that the consumer continued.

(f) Write the replay tool before you need it. Read the DLQ, print the reasons grouped by type, and re-produce selected messages to the main topic.

15.13 Summary

A Kafka topic is an append-only log, partitioned, read by position, and not consumed by reading. Four properties follow and they are the reasons to use it rather than a queue: replay (a consumer bug is fixable by resetting an offset), multiple independent consumers, ordering within a partition, and durability decoupled from consumption.

Partitions are the unit of parallelism, ordering, and assignment — three things at once, which is why partition count is the most consequential topic setting. The key decides the partition: same key, same partition, ordered; different keys, no guarantee. A null key round-robins and gives up ordering entirely. And changing the partition count breaks per-key ordering across the change, permanently — it is a decision with an ordering consequence, not a capacity knob.

Producers: acks=all with min.insync.replicas=2 and replication.factor=3 for anything that feeds a revenue number — and set the minimum explicitly, because acks=all with an in-sync set of one is acks=1 wearing a different name. enable.idempotence is nearly free and gives exactly-once to Kafka only. linger.ms=20 is a large throughput win for imperceptible latency. And produce() is asynchronous: a producer with no delivery callback silently drops failures, which is the most common producer bug.

Consumer groups assign each partition to exactly one consumer, so parallelism is capped by partition count, membership changes trigger rebalances, and different groups are independent. Commit after processing, not before — and enable.auto.commit=True is the default, committing on a timer with no knowledge of whether your processing succeeded. It is the setting that silently converts your pipeline to at-most-once.

Rebalancing is the failure you will meet first, and the two liveness mechanisms are the reason it is hard to diagnose: session.timeout.ms watches a background heartbeat, max.poll.interval.ms watches the poll loop, and it is the second that bites — a healthy process that is simply slow gets declared dead. The storm is self-sustaining, because reassignment gives the survivors more work. Three fixes in order: reduce max.poll.records (do this first), raise max.poll.interval.ms (treating the symptom, and it delays detection of genuinely stuck consumers), process asynchronously. Adding consumers does not help. And group.instance.id gives static membership, removing deploy-time churn for one line.

Retention deletes by age; compaction keeps the latest per key forever. Compaction is what makes a topic a table, which is why CDC topics are compacted — and the tombstone is what lets a key be removed. Kestrel uses compact,delete together on CDC topics, which is not obvious and is usually what you want. Retention is a recovery-window decision, not a cost decision: it bounds how long a consumer can be broken before recovery gets much harder — the same question as max_slot_wal_keep_size and vacuum retention, in three different systems.

Delivery semantics: at-least-once plus idempotent writes, with (topic, partition, offset) as an idempotency key that is unique by construction. Kafka transactions give exactly-once inside Kafka and nowhere else.

Sizing gives you a floor and a shape, not an answer. Little's Law on Kestrel's peak gives 8.7 handlers; multipliers for growth and catch-up suggest ~26; the topic has 12, because the consumer parallelizes internally, because 12 has convenient divisors, and because the peak lasts minutes. Choose a number with many divisors, size for two years, and remember that a consumer sized exactly for the arrival rate never catches up.

Dead-lettering is the right answer among crash, skip, and divert — and only if someone reads it. Four properties: the original bytes, the reason in headers, an alert on rate, and a replay tool written before you need it. A DLQ with no alert and no replay tool is a slower way to lose data: one held 2.1 million messages for fourteen months, and eight months of them aged out under a default retention nobody had configured. Set the DLQ's retention longer than the source topic's.

What's next

Chapter 16 is API ingestion — the least glamorous and most universally required skill in Part III. Rate limits expressed in four incompatible ways, pagination that changes shape at page 50, authentication that expires mid-job, retries that must distinguish "try again" from "stop immediately," and data that changes retroactively after you have read it.