Further Reading: Real-Time and Streaming Architecture

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This topic has one outstanding book and one outstanding paper, and reading them is worth more than any amount of framework documentation. The framework material teaches you an API; the two below teach you the model that every framework is an implementation of, which is what makes §29.4's three questions askable at all.

The two that matter

• Tyler Akidau, Slava Chernyak, and Reuven Lax, Streaming Systems (O'Reilly, 2018). The book on this subject. Chapters 1 through 4 are the core — event time versus processing time, windowing, watermarks, and the "what/where/when/how" framing that organizes everything else. §29.4 and §29.5 are a compression of it. Read it even if you use Spark or Kafka Streams rather than Beam; the model is the model. Tier 1.

• Akidau et al., "The Dataflow Model" (VLDB 2015). The paper the book grew out of, free, and about twelve pages. If you read one thing from this list, read this — it states the problem (unbounded, out-of-order, delayed data) and the solution (a small set of orthogonal questions) more crisply than anything since, and it is the origin of the vocabulary every framework now uses. Tier 1.

• Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 11. Streams from first principles: logs, change capture, exactly-once, and the relationship between a stream and a table. The treatment of "exactly-once" is the careful version of §29.6, and Kleppmann is precise about the scope in a way most framework documentation is not. Tier 1.

The primary sources

• The Apache Flink documentation on event time, watermarks, and idleness. §29.5's withIdleness is documented plainly and is easy to miss because it reads as an optimization rather than as a correctness setting. Read the watermark page and then the "Generating Watermarks" page, in that order. Tier 1 — versioned.

• The Flink documentation on state, savepoints, and operator UIDs. §29.7 and §29.10. The operator UID page is the one to read before your first deploy, not after it — it explains why a job that runs perfectly can be undeployable. Tier 1.

• The Spark Structured Streaming Programming Guide, particularly the output modes and watermarking sections. The output-mode discussion is genuinely useful even for non-Spark users, because it names a distinction (append versus update versus complete) that other frameworks leave implicit and that determines what a downstream consumer must handle. Tier 1.

• The Kafka documentation on transactions and exactly-once semantics, and the original KIP-98 design document. The KIP is more useful than the docs if you want to know what the guarantee actually covers, which is §29.6's whole point. Tier 2 — KIPs are archived and findable but the numbering is unfriendly.

The Lambda/Kappa material, read as history

• Nathan Marz and James Warren, Big Data (Manning, 2015) — the Lambda architecture, from its author. Read it as a historical document, which is the honest framing: it is a careful response to 2011's constraints, and §29.3 argues those constraints are gone. The reasoning is still worth seeing, because it shows what a serving layer has to reconcile and why. Tier 2 — dated, and deliberately so.

• Jay Kreps, "Questioning the Lambda Architecture" (2014). Short, free, and the counter-argument. Read both, in order, in an afternoon — and then notice that the resolution in §29.3 is neither. Tier 2 — a blog post on O'Reilly Radar; find the current URL.

• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013). Longer and better than the Lambda piece, and the durable one. The idea that a log is the primitive under both a stream and a table is what makes Chapter 14's CDC, Chapter 15's Kafka, and this chapter one subject rather than three. Tier 1 for the ideas; Tier 2 for the URL.

On testing

• The Flink TestHarness documentation, and Spark's MemoryStream. The real versions of code/stream_harness.py. Both are documented sparsely and used rarely, which is §29.9's complaint — the capability exists and almost nobody uses it, and the reason is that streaming tests feel like they need a cluster.

• Anything on property-based testing applied to event ordering. §29.9's second case — out-of-order arrival must give the same answer — is a property, and generating permutations of an event sequence and asserting invariance is a natural fit that almost nobody does. Hypothesis (Python) makes it about six lines. Tier 1 for Hypothesis's documentation.

On the request, rather than the technology

• Anything careful on requirements elicitation. Case Study 2's four questions are a small, specific instance of a general practice, and the requirements-engineering literature is better on it than the data literature is. The transferable idea is that a stated requirement is usually a proposed solution, and the work is recovering the need behind it. Tier 2 — a large literature; you need one chapter of it.

• Chapter 25 Case Study 1 and Chapter 26 §26.4 of this book. Case Study 2's review is the same move as the margin forecast and the paging list: make the consequence explicit and let the other person reach the conclusion. Three instances, three chapters, one technique.

Practice

• code/stream_harness.py in this chapter. Watermarks, tumbling and session windows, allowed lateness, per-partition watermarks, idleness, and state TTL — in two hundred lines with no dependencies. --demo idle reproduces Case Study 1 in one command, and its self-check contains §29.9's five cases.

• Measure your own processing_time − event_time distribution. Exercise 29.13. Everyone picks 30 seconds; almost nobody measures, and the p99.9 is usually a surprise.

• Run §29.11's four questions against the next "we need real time" request you receive. Case Study 2's team ran them eleven times and built the requested thing twice.

A note on what to be skeptical of

Any streaming benchmark or framework comparison. Chapter 11 §11.6's honesty checklist applies with extra force here, because streaming throughput numbers are almost always measured on a workload with no state, no joins, and no late data — which is to say, on the easy case, and the hard parts of this chapter contribute nothing to the number.

And "real-time" as a product description. It means sub-second in one document and sub-day in the next, and §29.1's question is the only reliable way to find out which. Asking "what latency, in seconds?" is a fair question that almost never gets a number, and the absence of a number is itself the answer.