Chapter 29 — Key Takeaways (Real-Time and Streaming Architecture)

The page to read before agreeing to build a streaming pipeline, and again before deploying one.

The claim

Streaming is not faster batch. It is a different set of correctness problems, and most of them follow from one fact: you must decide what to do about data that has not arrived yet. Batch waits until the window is over. Streaming cannot, so it needs a policy.

What latency is actually required

📐 "If this arrived thirty minutes later, what would go wrong?" Four answer clusters, and three do not need streaming:

| "Nothing, it just feels slow" | a scheduling problem with the current batch | | "We'd decide on stale data" | how often, and how stale? | | "A customer would see the wrong thing" | a specific surface, not the platform | | "An automated action would fire wrongly" | streaming |

Only a decision made per event, by code, with no human in the loop, is streaming.

The cost is not linear: nightly $19,000/yr · hourly micro-batch $23,000 · continuous $61,000, six extra weeks, and a new failure class on a four-person rotation. The step from hourly to continuous buys ~30 min → ~2 s, and buys nothing if no decision is made in that window.

Micro-batch is under-used: ~95% of the latency benefit at ~20% of the complexity, with Chapter 20's machinery unchanged.

Lambda and Kappa

🔎 Lambda's fatal flaw was maintaining the logic twice, and the two paths disagreeing. Kappa's was assuming the log holds everything. Neither won — stream processors got exactly-once and event-time semantics, table formats got ACID, micro-batch got good enough. The modern answer is one code path at whatever cadence the requirement needs.

What survives of Lambda is a technique, not an architecture: a fast approximate answer plus a slow correct one. A cached dashboard with a nightly reconciliation is exactly that.

Watermarks

Event time is when it happened; processing time is when you saw it. A watermark asserts "I believe I have seen everything before T" — always a heuristic.

⚠️ It forces three questions, and the default answers all three in the worst way (drop silently):

1. How long do I wait? Measure processing_time − event_time and choose from the tail — Kestrel's clickstream: p99 4.1 s, p99.9 28 s, max 16 min.
2. What happens to late data? Drop (and count it) · emit a correction · side-output.
3. What do consumers do with a correction? ← not a streaming question. It is a Chapter 17 contract question, and a consumer that cannot handle a restatement double-counts.

Windows

Tumbling · sliding · session. A one-hour window sliding every minute holds each event in sixty windows — the commonest cause of unexpected state size.

⚠️ A watermark across partitions is the MINIMUM. One idle partition stalls emission for everything, and the job looks completely healthy. with_idleness is one line.

It is not free: it trades a stall for a probability of dropped late data. Right when timeliness matters more than completeness; wrong when it does not — and write that reasoning next to the code, including the instruction not to copy it.

Delivery guarantees

🎓 "Exactly-once delivery is impossible — you cannot distinguish a lost message from a lost acknowledgment. Exactly-once processing means the effects appear once." Achieved by a transactional sink, or 🔁 an idempotent one — Chapter 20 §20.3, unchanged, and usually simpler.

The guarantee ends at the sink's boundary: external API calls · non-participating sinks · anything downstream. For any side effect leaving the system, carry a deduplication key.

State

How much · where it lives · how it is checkpointed · ⚠️ when it expires. A dedup set with no TTL dies in three months with an unattributable OOM. 26 hours, not 24 — a TTL equal to the window drops keys at the boundary.

And state is not queryable. A batch job's intermediates are tables; a streaming job's state is inside the job. Emit it as a side output where you can afford to.

Joins

📐 A join window is a business decision, not a tuning parameter. A watermark's delay describes your infrastructure; a join window describes what the business permits.

Kestrel: payment lag p99.9 42 seconds, business answer three days — which established it was not a streaming join at all. When the business window and the affordable state window differ by orders of magnitude, you need both paths — Lambda's surviving technique, applied to one join.

A stream–stream join's time bound is not a filter; it is what makes the state bounded.

Testing

🧪 Streaming is deterministically testable, because event time is data. advance_watermark_to is the whole technique — no clocks, no sleeps, no flakiness.

Five cases; the last three are almost never written: happy path · out-of-order arrival, same answer · a late event inside allowed lateness (does the correction do what you meant?) · one outside it — is the drop COUNTED? · a restart mid-window, which catches missing operator UIDs in a test rather than a deploy.

Operating

Four failure modes batch does not have: consumer lag (alert on the ratio, not a threshold) · backpressure (the system working) · checkpoint age (an absence) · and a restart that resumes mid-window with state.

🏭 Deploying a stateful job has no batch equivalent. Savepoint, restore, and the new job must read the old state — so set explicit operator UIDs from version one. Chapter 27's deploy shape gains a fourth value: does this change the state schema? If yes, run the new job in parallel from the log and cut over — Kappa's one durable contribution.

The two case studies

Three weeks of nothing, every night. One quiet overnight partition stalled the global watermark for 2h 43m a night. Consumer lag was zero, checkpoints succeeded, backpressure was none, CPU was 12% — and the only check that would have caught it was a four-line freshness query on the sink.

Correctness and timeliness are separate properties. The alerts were correct, arrived hours late, and every one of Chapter 23's twenty-two assertions passed. The register gained a twenty-third: max(emission_time − event_time), because the other twenty-two were written for batch, where the schedule makes latency a known constant.

The real-time requirement that was a scheduling bug. Six weeks and $38,000 estimated; the fix was a cron expression and took eleven minutes. A decision made at a fixed time cannot use data fresher than the last refresh before it.

⚠️ A cron offset is a dependency written in a form nothing can check, and it decays silently when the upstream moves. No test fails; no alert fires; the person who moved it had no way to know.

🏭 Four questions in front of every "we need X": what decision, by whom, how often · what breaks at +30 minutes · is the current thing meeting its own stated requirement ← the most-skipped, and it found the bug · and only then, what would we build. Of eleven reviews, six found a problem with something existing and two produced the requested build.

Fix the actual problem first, then discuss the request. Show the arithmetic. Name the case where they would be right. The complaint was correct and only the solution was wrong — separating those is most of the skill.