Further Reading: Event Streaming with Apache Kafka

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Kafka has the best documentation of any system in this book, and the best of it is the configuration reference — which nobody reads and which contains the answer to most operational questions.

The primary sources

• The Apache Kafka documentation (kafka.apache.org/documentation). Read three parts properly: "Design" (the log abstraction, replication, and the reasoning behind it — this is §15.1 from the source), the producer configuration reference, and the consumer configuration reference.

The configuration references are unglamorous and are the highest-value pages in this list. max.poll.interval.ms, max.poll.records, enable.auto.commit, acks, min.insync.replicas, and group.instance.id are all documented with their interactions stated, and every incident in this chapter is a case of one of those interactions. Tier 1.

• KIP-345 (static membership) and KIP-429 (incremental cooperative rebalancing). Kafka Improvement Proposals are the design documents, and these two explain the rebalance behavior in §15.5 better than the user documentation does — including why a rebalance stops the whole group and what cooperative rebalancing changes about that. Read KIP-429 if rebalancing is costing you anything. Tier 1.

• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013). Recommended in Chapters 1 and 14, and this is the chapter it is most directly about. It is the clearest explanation of why Kafka is shaped the way it is, and reading it makes the rest of the documentation feel inevitable rather than arbitrary. Tier 2 — has moved more than once.

The books

• Gwen Shapira, Todd Palino, Rajini Sivaram, and Krit Petty, Kafka: The Definitive Guide, 2nd edition (O'Reilly, 2021). The standard reference. Chapters 3 and 4 (producers and consumers) cover §15.3 through §15.5 at four times this chapter's length, and Chapter 6 on reliable data delivery is the proper treatment of §15.7. Check the edition — the 2nd covers KRaft and the modern consumer, the 1st does not. Tier 1.

• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 11 ("Stream Processing"). The vendor-neutral framing: what a log-based message broker is, how it differs from AMQP/JMS-style brokers, and why the difference matters. Read it alongside the Kafka documentation to separate what is essential from what is Kafka's choice. Tier 1.

On delivery semantics

• Neha Narkhede, "Exactly-once Semantics are Possible: Here's How Kafka Does it" (2017). Recommended in Chapter 4 and relevant again. Read the body, which is honest about the transaction boundary, and read the title as an object lesson in how the claim gets transmitted. §15.7's scepticism is aimed at the transmission, not the engineering. Tier 2.

• KIP-98 (exactly-once delivery and transactional messaging). The design document behind the transactional producer. Longer and more precise than any blog post about it, and it states the boundary explicitly. Tier 1.

On operating it

• Confluent's and AWS MSK's documentation on monitoring. Both publish the metrics that matter — consumer lag, under-replicated partitions, offline partitions, request latency — and the thresholds they recommend. Vendor material, useful for the metric names and the reasoning, and read the thresholds as starting points. Tier 2.

• Burrow (LinkedIn's consumer lag monitoring tool) and its design rationale. Burrow's argument — that a lag threshold is the wrong alert and that you should evaluate lag trends per partition — is exactly this chapter's first case study, formalized and productized. Read the rationale even if you use something else. Tier 2.

• Any write-up of a Kafka rebalance incident. They are numerous, they are consistent, and the consistency is the finding. Search for "kafka rebalance storm max.poll.interval" and read three from different organizations. Tier 2 — blog posts, quality varies.

On the alternatives

• The Redpanda documentation, particularly on Kafka API compatibility and on what differs. Worth knowing because Redpanda is a drop-in for the Kafka protocol with a different implementation (no JVM, no ZooKeeper), and it is what many small teams should use for local development. Tier 2 — actively changing.

• Apache Pulsar's documentation on its segment-based architecture, read as a comparison. Pulsar separates serving from storage in a way Kafka does not, which changes the operational profile around partition counts and scaling. Useful for understanding which of Kafka's constraints are essential and which are implementation choices. Tier 2.

• The AWS Kinesis and Google Pub/Sub documentation, if you are on those platforms. Both solve the same problem with different guarantees — Kinesis's shard model is close to partitions, Pub/Sub's is not — and reading either after this chapter is a good test of whether you learned the concepts or the product. Tier 2.

If you only read one thing

Read the Kafka consumer configuration reference, top to bottom, once. It is perhaps forty minutes.

It is genuinely dull and it contains the direct explanation of every failure in this chapter: max.poll.interval.ms and max.poll.records and their interaction, enable.auto.commit and what it actually does, group.instance.id, and auto.offset.reset — which this chapter did not have room for and which will surprise you the first time a consumer group is deleted.

Then, before your next Kafka incident, put this in your runbook above the dashboards:

kubectl logs -l app=<consumer> --since=15m \
  | grep -iE 'rebalanc|leavegroup|poll timeout|coordinator'

The client's own log messages are more specific than anything you have instrumented, because the people who wrote them knew exactly which mistakes you were going to make.