Chapter 29: Real-Time and Streaming Architecture

"They asked for real time. What they meant was that they had been waiting until eleven the next morning, and they wanted it before their nine o'clock meeting."

Overview

Chapter 15 built a Kafka consumer. This chapter is about whether you should have one, and about what changes in every other part of this book when the answer is yes.

The material has a reputation for being about technology — Flink versus Spark Streaming versus ksqlDB — and that is the least important part of it. The hard parts of streaming are three questions that no tool answers for you: what latency is actually needed, what happens to data that arrives late, and what "exactly once" means when the thing you are writing to is a table someone else is reading.

Chapter 3 §3.2 said this chapter is written for the case where an automated system acts without a human; §29.1 makes that test operational, and §29.3 settles the Lambda/Kappa question it left open. Chapter 4 §4.6 and Chapter 20 §20.7 both promised watermarks done properly; that is §29.4.

The chapter's organizing claim:

Streaming is not faster batch. It is a different set of correctness problems, and most of them are consequences of one fact: you must decide what to do about data that has not arrived yet. Batch answers that by waiting until the window is over. Streaming cannot, so it has to have a policy.

29.1 What Latency Is Actually Required

The most valuable section in this chapter, and it is a conversation rather than a technique.

"Real time" is almost never a requirement. It is a shorthand for "sooner than now", and the useful question is the one that makes the requirement concrete:

What decision is made on this data, how often is it made, and what does being late cost?

Four answers, and they lead to four completely different architectures:

Only the last one is streaming, and it is streaming because a human is not in the loop — the decision is made by code, at the moment of the event, and there is nothing to wait for.

📐 Design Decision — the question that resolves most "we need real time" requests

Ask: "if this arrived thirty minutes later, what would go wrong?"

The answers cluster, and three of the four clusters do not need streaming:

• "Nothing, but it feels slow." → a latency problem with the current batch, not a case for streaming. Kestrel's dashboard was stale until 11:00 because a job was scheduled badly, and moving it to 03:00 solved the entire request.
• "We'd make a decision on stale data." → how often, and how stale? A decision made twice a day needs hourly data at most.
• "A customer would see the wrong thing." → now we are talking, and the requirement is on a specific surface, not on the whole platform.
• "An automated action would fire wrongly." → streaming, and the latency requirement is whatever the action's window is.

The reason to push on this is that the cost is not linear. Kestrel's estimate, and the shape generalizes:

	Build	Run/year	On-call
nightly batch	2 weeks	$19,000	the existing rotation
hourly micro-batch	+3 days	$23,000	the same
continuous streaming	+6 weeks	$61,000	a new failure class, 24/7

The step from hourly to continuous is where the cost is, and it buys latency from ~30 minutes to ~2 seconds. If no decision is made in that window, it buys nothing — and Chapter 26 §26.1's four-person rotation is the constraint that makes the third row genuinely expensive.

29.2 The Spectrum

Batch and streaming are not two things. They are the ends of a continuum, and most of the useful positions are in the middle.

    nightly      hourly      micro-batch     continuous      per-event
    batch        batch       (1-15 min)      streaming       processing
      │            │              │              │               │
   ~12 hrs      ~1 hr        ~5 min          ~2 sec          ~50 ms
      │            │              │              │               │
   cheapest    cheap        moderate         expensive      most expensive
   simplest    simple       moderate         complex        complex

Four things change as you move right, and only the first is the one people think about:

Latency falls, which is the point.

Correctness gets harder. Every window you close is a decision made without knowing what arrives next. §29.4.

Operations get harder. A batch job that fails is re-run; a streaming job that fails has state, and Chapter 26's runbook has to cover a new class of failure.

Cost rises non-linearly, because a continuous job holds resources continuously.

The middle of the spectrum is under-used, and the most common good answer is micro-batch: a job every five minutes, using ordinary batch tools. It is 95% of the latency benefit at 20% of the complexity, and Chapter 20's incremental machinery works unchanged.

29.3 Lambda and Kappa, With Hindsight

Lambda (Nathan Marz, ~2011) proposed running two paths: a batch layer computing correct results slowly over all history, and a speed layer computing approximate results quickly over recent data, with a serving layer merging them.

It solved a real problem. In 2011 stream processors could not do exactly-once, could not handle late data, and could not reprocess history — so the batch layer was there to correct the stream.

Kappa (Jay Kreps, ~2014) was the counter-proposal: one path. Keep everything in a replayable log; when you need to reprocess, replay the log through a new version of the streaming job.

With a decade of hindsight, here is what actually happened, and it is neither:

🔎 Read the Plan — what the debate was really about, and how it resolved

Lambda's fatal problem was never architectural. It was that you maintain the same business logic twice, in two languages, on two engines, with two sets of bugs — and the two paths disagree, which produces a class of incident where nobody can say which number is right.

Kappa's fatal problem was that it assumed the log holds everything. Replaying two years through a streaming job requires two years in the log (expensive), a job that can process it faster than it accumulated (usually), and a sink that tolerates the rewrite. For most organizations that was not true in 2014 and is not true now.

What replaced both is not a third architecture. It is a set of capabilities that made the question less interesting:

• Stream processors got exactly-once and event-time semantics (Flink, then Spark Structured Streaming, then Beam's model everywhere). The speed layer stopped being approximate, which removed Lambda's whole reason for existing.
• Table formats got ACID and time travel (Chapter 10). A streaming job can now write to the same table a batch job reads and rewrites.
• Micro-batch got good enough that the latency gap between "batch" and "streaming" narrowed to a region most requirements do not live in.

So the modern answer is: one code path, on a table format that supports both, at whatever cadence the requirement needs. Which looks like Kappa and is really just "do not write your logic twice."

And the part of Lambda that survives is worth naming, because it comes back under other names: a fast approximate answer plus a slow correct one is a genuinely useful pattern — it is what a cached dashboard with a nightly reconciliation is, and what Chapter 18 Case Study 2's shadow model is. The mistake was making it the architecture rather than a technique.

29.4 Event Time, Processing Time, and Watermarks

The central idea in stream processing, and the one Chapter 20's INTERVAL '3 days' was a crude approximation of.

Event time is when the thing happened. Processing time is when your system saw it. They differ, by milliseconds usually and by hours occasionally, and every hard problem in this chapter comes from that gap.

event time    ──●───────●──────●──●────────────●──▶
                 \       \      \  \            \
processing    ────●───────●──────●──●────●───────●──▶
time                                     ↑
                                    arrived late:
                                    event time was earlier

A watermark is an assertion: "I believe I have seen all events with an event time before T."

It is a heuristic, always, because you cannot know. And it is the mechanism by which a stream processor decides that a window is closed and can be emitted.

Two parameters, and they are the whole policy:

The watermark strategy — usually "the maximum event time seen, minus a bounded delay." A delay of 30 seconds says: I will wait 30 seconds past the newest event before declaring an earlier window complete.

Allowed lateness — how long after a window closes you will still accept and re-emit for it.

# Flink, and the shape is the same in Spark and Beam.
WatermarkStrategy
    .for_bounded_out_of_orderness(Duration.of_seconds(30))
    .with_idleness(Duration.of_minutes(1))          # ← §29.5

⚠️ Failure Mode — the three questions a watermark forces you to answer, whether or not you do

A batch job answers these by waiting until the day is over. A streaming job answers them at every window boundary, and if you do not choose, the default chooses.

1. How long do I wait? Too short and you close windows before the data arrives, producing results that are wrong and then corrected — or wrong and never corrected. Too long and you have given back the latency you built this for.

Measure it. The distribution of processing_time − event_time over a fortnight, and pick from the tail — Chapter 20 §20.7's lesson, restated: the events you drop are in the tail by definition. Kestrel's clickstream:

text p50 0.4 s p99 4.1 s p99.9 28 s ← mobile clients with poor connectivity max 16 min ← an app that batches events while offline

A 30-second watermark drops the p99.9 tail and everything beyond it.

2. What happens to what arrives after? Three options, and you must pick one: drop it (fastest, and it must be counted), emit a correction (correct, and every downstream consumer must handle an update), or route it to a side output for separate handling.

3. What do downstream consumers do with a correction? This is the question that makes streaming genuinely harder than batch, and it is not a streaming question at all — it is a contract question (Chapter 17), and a consumer that cannot handle a restatement will silently double-count.

The default in most frameworks is to drop late events silently, which answers all three questions in the worst way available and produces a number that is quietly low forever.

29.5 Windowing

Three shapes, and choosing between them is usually easy once you have the vocabulary:

Tumbling. Fixed, non-overlapping. Revenue per five-minute bucket. Each event belongs to exactly one window.

Sliding. Fixed length, overlapping. Revenue in the last hour, updated every minute. Each event belongs to many windows, which multiplies state — a one-hour window sliding every minute means each event is held in sixty windows at once, and that factor is the single most common cause of a streaming job's state being larger than anyone predicted.

Session. Gap-based. Chapter 18 §18.9's sessionization, and the streaming version has exactly the same boundary problem — solved here by the watermark rather than by an overlap read.

⚠️ Failure Mode — an idle partition stalls the watermark for everything

The subtlest operational failure in this chapter, and it is not intuitive.

A watermark across parallel partitions is the minimum of the per-partition watermarks, because a window cannot be complete until every partition has passed it.

So one partition with no traffic holds the whole watermark back, and windows stop emitting — for every partition — while everything looks healthy: no lag, no errors, no backpressure. The job is running and producing nothing.

Kestrel met this on the clickstream topic, which is partitioned by anonymous_id. At 04:00 UTC the traffic drops enough that some partitions go quiet for minutes, and the aggregation stopped emitting between about 03:40 and 06:20 every night for three weeks before anyone connected the gap to the idle partitions.

The fix is one line, and every framework has it:

python .with_idleness(Duration.of_minutes(1)) # a quiet partition stops holding the watermark

And the monitoring lesson is Chapter 25 §25.9's, in a new place: this is an absence. No metric the job emits shows a problem, because the job is fine. The signal is the output, and the check is a freshness assertion on the sink — which is the one thing that would have caught it and is the last thing anyone instruments on a streaming job.

29.6 Delivery Guarantees, and What Exactly-Once Means

Three guarantees, and the third is widely misunderstood:

At-most-once. Fire and forget. Fast, lossy, and correct for genuinely disposable telemetry.

At-least-once. Retry until acknowledged. Duplicates are guaranteed, and this is the default in most systems.

Exactly-once. And here is the honest statement:

"Exactly-once delivery" is impossible in a distributed system. "Exactly-once processing" is achievable, and it means: the effects of processing each record appear exactly once, even though the record may be delivered many times.

Which is achieved in one of two ways, and neither is magic:

A transactional sink. The processor writes its output and its offset commit in one atomic transaction. Kafka's transactions, Flink's two-phase commit sinks, Delta's atomic commits.

An idempotent sink. Writing the same record twice has the same effect as writing it once — which is Chapter 20 §20.3's four strategies, arriving in a streaming context unchanged.

The second is usually simpler and is under-used. A merge on a natural key, or a partition overwrite, gives you exactly-once effects without any transactional coordination at all.

🔁 Idempotency Check — exactly-once ends at your sink's boundary

The guarantee covers the processor's own state and its writes to a sink that participates in the protocol. It stops there, and three things sit outside it:

An external API call. A webhook, a payment, an email. These are the ones that hurt — a re-processed record sends the message twice, and the protocol cannot help because the remote system is not in the transaction.

A sink that does not participate. Writing to a plain S3 prefix, a REST endpoint, a legacy database. The framework will still say "exactly-once" because it is describing its own checkpointing.

Anything downstream of the sink that is not itself idempotent — which is Chapter 20's whole chapter and Chapter 27 §27.8's fourth rollback category.

The practical rule: for any side effect leaving your system, carry a deduplication key and check it, and treat "exactly-once" as a property of a path rather than of a job:

```python

The only thing that makes an external call safe under retry.

if not seen.check_and_set(record.idempotency_key): send_webhook(record) ```

And ask your framework's documentation which sinks participate. The list is shorter than the marketing implies, and it is the difference between a guarantee and a hope.

🎓 Interview Angle — "explain exactly-once"

A very common question, and the answer that separates people who have operated a streaming system from people who have read about one is where the guarantee stops.

A weak answer describes the checkpoint protocol. It is usually correct and it is a recitation.

A strong answer is three sentences:

"Exactly-once delivery is impossible — you cannot distinguish a lost message from a lost acknowledgment. What is achievable is exactly-once processing: the effects of each record appear once, even though the record may be delivered many times. That is done either with a transactional sink, or by making the sink idempotent — and the second is usually simpler."

Then the part that demonstrates operational experience:

"And the guarantee ends at the sink's boundary. If the job calls an external API, sends an email, or writes to something that does not participate in the protocol, exactly-once does not cover it — the framework is describing its own checkpointing. For anything leaving the system you carry a deduplication key."

A likely follow-up is "so is exactly-once a marketing term?" — and the honest answer is no: it is a precise claim about a specific scope, and the problem is that the scope is rarely stated. Which is the same shape as Chapter 27's "do not X; instead Y": a true statement whose omitted half is where the failures are.

29.7 State, and What It Costs

A stateless streaming job is easy. Filter, map, route — each record independent, failure recovery trivial, and this is a large fraction of real streaming work.

A stateful job holds things: window contents, aggregations, join buffers, deduplication sets. And state is where the operational difficulty lives.

Four questions to ask of any stateful streaming job:

How much state? Sessionizing 700,000 daily sessions with a 30-minute gap means roughly 7,000 open sessions at any instant (Chapter 18 Case Study 2's arithmetic) — small. Deduplicating 14,000,000 daily events over a 24-hour window means 14 million keys — not small.

Where does it live? In-memory is fast and bounded by the heap. RocksDB spills to local disk and handles far more. The choice is usually forced by the first question.

How is it checkpointed? Periodically, to durable storage. Checkpoint interval is a direct trade: shorter means less reprocessing on failure and more steady-state overhead.

When does it expire? ⚠️ The one people forget, and the failure is slow:

# Without a TTL, a deduplication set grows forever and the job dies in
# three months with an out-of-memory error nobody can attribute.
state_ttl = StateTtlConfig.new_builder(Time.hours(26)).build()

26 hours, not 24, because the window is 24 and you want margin — and because a TTL exactly equal to the window drops keys at the boundary.

And state has a fifth property that only shows up in an incident: it is not queryable. A batch job's intermediate results are tables you can look at; a streaming job's state is inside the job, in RocksDB on a task manager, and answering "why did this session not close?" means either a state processor API, a debug log added and redeployed, or a reasoned guess.

Which argues for emitting state as data where you can afford to. Kestrel's sessionizer writes a side output of currently-open sessions every minute — a few thousand rows, negligible cost — and it converted the two incidents it has had from archaeology into a query. Chapter 24 Case Study 1's _ingested_at argument, in a place where it is much harder to add afterwards.

29.8 Joins in a Stream

Joining is where streaming complexity actually lives, and it is under-covered relative to windowing because windowing is easier to draw.

The difficulty is that a join asks two questions a stream cannot answer directly: has the matching record arrived? and will it ever?

Three kinds, in increasing order of difficulty:

Stream–table (enrichment). A stream of events joined to a mostly-static dimension. The common case, and the easy one — Kestrel's clickstream joined to dim_product.

events.join(broadcast(dim_product), "sku")     # the table is small and replicated

⚠️ And it has one trap, which is Chapter 20 §20.11's late-arriving dimension in a new setting: when the dimension is updated by its own stream, the join is temporal — you want the dimension row as it was at the event's time, not as it is now. Frameworks call this a temporal join or a versioned table join, and using an ordinary join instead means replaying history produces different answers than the original run did, which is the property that makes a replay useless.

Stream–stream, windowed. Two streams, matched within a time bound. An order event and its payment event, within five minutes.

SELECT o.order_id, p.amount_cents
  FROM orders o JOIN payments p
    ON o.order_id = p.order_id
   AND p.event_ts BETWEEN o.event_ts AND o.event_ts + INTERVAL '5' MINUTE

The BETWEEN is not optional and is not a filter. It is what makes the join's state bounded: without a time bound, the processor must keep every record from both sides forever, and most frameworks will refuse the query rather than let you find that out in production.

Stream–stream, unbounded. Every order, matched to its payment, whenever it arrives. This is not a streaming problem; it is a stateful problem wearing streaming clothes, and the honest answer is usually to land both streams and join them in batch.

📐 Design Decision — the join window is a business decision, not a tuning parameter

Kestrel needed to match order events to payment events. The engineer's first instinct was to size the window from the observed distribution — which is exactly right for a watermark (§29.4) and exactly wrong here.

The difference: a watermark's delay is a statement about your infrastructure — how late does data arrive. A join window is a statement about the business — how long after an order can a payment legitimately arrive?

text observed p99.9 payment lag 42 seconds observed maximum 11 minutes the business answer up to 3 DAYS

Three days, because a payment can be retried, a card can be re-presented, and a bank transfer takes what it takes. A five-minute window would have silently dropped every one of those, and they are disproportionately the interesting ones — the failed payments.

And three days of two-sided join state is not viable in memory, which is the useful outcome of asking the question: it establishes that this is not a streaming join at all. Kestrel emits matched pairs within five minutes for the operational dashboard, and routes the unmatched to a batch reconciliation that runs hourly against both landed streams.

The general shape: when the business window and the affordable state window differ by orders of magnitude, you need both paths — and that is Lambda's surviving technique (§29.3), applied deliberately to one join rather than to an architecture.

29.9 Testing a Streaming Job

"Run it and watch" is not a test, and it is what most streaming jobs have, because testing something continuous feels like it needs something continuous.

It does not. The key insight is that event time is data, so a test can drive it:

# A deterministic test. No clocks, no sleeps, no flakiness.
harness = TestHarness(sessionize, watermark_delay=Duration.of_seconds(30))

harness.process(Event(anon="a", ts="10:00:00"))
harness.process(Event(anon="a", ts="10:00:20"))
harness.advance_watermark_to("10:31:00")        # ← the whole technique

assert harness.output() == [Session(anon="a", start="10:00:00", n=2)]

advance_watermark_to is what makes streaming testable. You control time, so the test is deterministic, runs in milliseconds, and can exercise cases that would take hours to produce naturally.

Five cases every stateful streaming job's tests should contain, and the last three are the ones that are almost never written:

The happy path. Events in order, window closes, correct output.

Out-of-order arrival. Events delivered in a different order than their event times, within the watermark. Must produce the same answer.

A late event, inside allowed lateness. Does it produce a correction, and is the correction what you intended?

A late event, outside allowed lateness. ⚠️ Is it dropped, and is the drop counted? A test that asserts the event is dropped and does not assert the counter incremented is a test that passes on a job with no counter.

A restart mid-window. Take a savepoint, restore, continue. Does the window still close correctly? This is the case that catches missing operator UIDs (§29.10), and it catches them in a test rather than during a deploy.

🧪 Try It — the twenty-minute test that would have caught §29.5's idle partition

The idle-partition stall ran for three weeks and produced no signal on any of the job's own metrics. It is trivially testable, and the test is worth writing for any partitioned streaming job:

```python harness = TestHarness(job, parallelism=2)

Partition 0 receives traffic. Partition 1 receives nothing.

harness.process(Event(key="k0", ts="10:00:00"), partition=0) harness.process(Event(key="k0", ts="10:05:00"), partition=0)

Without with_idleness, the watermark is min(p0, p1) and p1 has none.

assert harness.output() == [] # ← the bug, asserted ```

Then add with_idleness and assert the output appears. Two assertions, and they encode a three-week incident.

The generalizable move: for any operational failure you have had, ask whether a test harness that controls time could have reproduced it. For streaming the answer is usually yes, because nearly every streaming failure is about when something arrived — and when is a parameter.

29.10 Operating a Streaming Pipeline

Everything in Chapters 24 through 28 applies, plus four failure modes batch does not have.

Consumer lag. The gap between the latest offset and your committed one. The single most important streaming metric, and the one to alert on — with Chapter 25 §25.4's rule: the ratio to a trailing median, not an absolute threshold, because acceptable lag is workload-specific.

Backpressure. A downstream stage cannot keep up, so upstream slows. This is the system working correctly, and it is worth alerting on because sustained backpressure means under-provisioning.

Checkpoint failures. If checkpoints stop succeeding, the job is running and cannot recover — a restart replays from the last successful one, which may be hours ago. Alert on checkpoint age, which is another absence (Chapter 25 §25.9).

A restart that is not a restart. A batch job restarted is a batch job re-run. A streaming job restarted from a checkpoint resumes mid-window with state, and a job restarted without state is a different program producing different output.

🏭 From the Pipeline — the deploy that is not a deploy

Deploying a change to a stateful streaming job is the operation that has no batch equivalent, and it is where Chapter 27's material meets its hardest case.

Stopping and starting loses state, unless you take a savepoint — an explicit, on-demand checkpoint you restore from:

bash flink stop --savepointPath s3://kestrel/savepoints my-job-id flink run --fromSavepoint s3://kestrel/savepoints/savepoint-abc123 new.jar

And the new job must be able to read the old state, which means:

• Adding an operator is usually fine. Removing one, or reordering, is not, unless every operator has a stable explicit UID — which almost nobody sets until the first time they cannot deploy.
• Changing a state type is a migration, with all of Chapter 17's compatibility rules.
• Changing parallelism is supported and reshuffles state, which is slow at scale.

The practice worth adopting before you need it: set explicit UIDs on every stateful operator, from the first version. It costs one argument per operator and it is the difference between a deploy and a rebuild-from-scratch.

And Chapter 27 §27.7's deploy shape has a fourth value for streaming jobs: does this change the state schema? If yes, the deploy is a migration, and the fallback — run the new job in parallel from the log and cut over — is Kappa's one genuinely durable contribution.

29.11 When Streaming Is Actually Right

Four situations, and the honest observation is that most data teams have one or two, not many:

A decision is made per event, by code. Fraud, personalization, routing, alerting.

The volume makes batch impractical. At some rate, "process the day's data" stops fitting in a day. Kestrel's 14,000,000 events a day does not reach this.

The source is a stream and there is no batch alternative. IoT, CDC (Chapter 14), some vendor APIs. Note that this argues for streaming ingestion, not streaming transformation — landing a stream and processing in micro-batches is entirely normal.

The value of the data decays in seconds. An operational dashboard someone acts on in the moment.

And the honest counter-list, of reasons that appear often and are not sufficient:

"Our competitors have real-time." Possibly, for one surface. "The data arrives continuously." So does everyone's. That is an ingestion property. "It would be more elegant." It would be more complex. "Executives asked for real time." §29.1's question.

💸 Cost Check — what streaming costs against the batch it replaces

The comparison people make is per-record and it is the wrong one, because a streaming job's cost is dominated by being up.

```text THE SAME AGGREGATION, two ways

BATCH, hourly micro-batch 24 runs x 3 min x 4 nodes x $2.400/node-h $28.80 / day plus warehouse: negligible $864 / month

STREAMING, always on 4 nodes x 24 h x $2.400 $230.40 / day plus a checkpoint store, and its requests ~$4 / day $7,032 / month ───────────── the ratio 8.1x ```

Eight times the cost for latency that one of §29.1's four clusters actually needs. That is the arithmetic behind the chapter's opening argument, and it is worth having in a spreadsheet before the conversation rather than during it.

Three things the table understates.

The streaming cluster cannot be scaled to zero. A batch job's cost is proportional to work; a streaming job's is proportional to time, so a quiet Sunday costs exactly what Black Friday costs.

State is a cost that grows. A windowed aggregation over anonymous_id holds state proportional to the key space, and checkpointing it every minute is a write of that state to durable storage. At 1.4 million sessions a day the checkpoint is not negligible and it is not in most estimates.

And the operational cost is the largest line and is not on this page. Chapter 29 §29.10's four failure modes — lag, backpressure, checkpoint failures, and a restart that loses state — are four things a batch pipeline does not have, and they are paid in engineer attention on a four-person team (Chapter 5 §5.1).

The honest framing for a proposal: streaming costs about 8× the compute and about 25% of an engineer. When the decision genuinely needs sub-second latency, that is a bargain. For the other three clusters it is a very expensive way to make a dashboard feel modern.

📏 Scale Note — state is the thing that scales badly

A stateless streaming job scales like a batch job. A stateful one does not, and the difference is the whole operational subject of this chapter.

text operation state held scales with ───────────────────────────────────────────────────────────────────────── filter, map, project none nothing. Free. a 5-minute tumbling window one entry per key per window ACTIVE KEYS a session window one entry per open session open sessions, (30 min gap) and a long tail a stream-stream join both sides, for the window the WIDER window deduplication by event_id every id seen, for the TTL the TTL x rate

The bottom row is the one that surprises people. A dedup set over Kestrel's clickstream with a 24-hour TTL holds 14 million ids; at 32 bytes each that is ~450 MB of state per operator instance, checkpointed every minute. Extend the TTL to seven days "to be safe" and it is 3.1 GB, and the checkpoint duration goes up with it.

Three consequences worth designing around:

Every state store needs an explicit TTL, and the TTL is a correctness decision rather than a memory one — it is exactly how late a duplicate can arrive and still be caught. Setting it from memory pressure gets the wrong answer in both directions.

Session windows have a long tail by construction. Most sessions close in minutes; a browser left open holds one for hours. A session-window job's state is sized by the tail, not by the median, and the tail is the thing nobody measures.

And a stream-stream join's state is the sum of both sides for the join window (§29.9), which is why the window is a business decision with a memory bill attached.

The number to watch is checkpoint duration, not state size. When a checkpoint takes longer than the interval between checkpoints, the job is no longer keeping up and the symptom is backpressure attributed to the sink.

🔐 Privacy & Governance — state is a copy, and checkpoints are backups of it

A stateful streaming job holds personal data in three places that no catalog knows about.

text where what lifetime ───────────────────────────────────────────────────────────────────────── operator state, in memory open windows, dedup sets, until the TTL, join buffers or forever the state backend, on disk the same, spilled the job's lifetime CHECKPOINTS, in object a full copy of state, until they are storage written every minute pruned, and the retention is a setting

The checkpoint row is the one that matters. A checkpoint is a durable, complete copy of the job's state — including a dedup set keyed by event_id for every event in the last 24 hours, and every open session's buffered events. It is written to object storage, it is retained by count or by age, and it is not in any catalog, any deletion manifest, or any access review.

Three consequences:

An erasure request does not reach state. Deleting a person from the warehouse leaves their events in the job's open windows and in every retained checkpoint. The state's TTL is the deletion latency, and the checkpoint retention is a second, longer one.

Checkpoints must be classified and retained deliberately. A default of "keep the last 20" on a minute-level checkpoint is twenty minutes; on an hourly one it is most of a day. Neither is unreasonable and neither was decided.

And the state's TTL is a privacy setting as well as a memory one (see the 📏 above). A dedup TTL of seven days "to be safe" is seven days of retained identifiers, chosen for correctness and paid for in obligation.

The practical instruction, and it costs one line in the ADR: state the state. "This job holds event ids for 24 hours and open sessions for up to 12; checkpoints are retained for 1 hour; both are in the deletion manifest." Nobody writes that sentence, and it is the only record that will ever exist of where the data is.

🧭 Version Note — the streaming/batch boundary keeps moving toward batch

Every year, something that required a streaming engine becomes a configuration option on a batch one, and the trend has been in one direction for a decade.

text capability used to require now ───────────────────────────────────────────────────────────────────────── minute-level freshness a streaming job a micro-batch, or a warehouse task every few minutes incremental, exactly-once table updates a stream + a state store a table format's MERGE late-data handling a watermark a lookback window (ch 20 §20.7) continuous ingestion from object storage a custom watcher auto-loading file notification services streaming SQL a specialised engine materialized views that refresh incrementally

The consequence for §29.1's argument is that it gets stronger over time, not weaker: the latency at which batch stops being sufficient keeps falling, so the fourth cluster shrinks.

What has not moved and will not: anything where a decision is made by code at the moment of the event. Fraud checks, ad bidding, real-time personalisation. There is no batch version of "block this transaction now", and that is the whole of the remaining category.

Two practical implications.

Re-ask the question periodically. A streaming job built three years ago for minute-level freshness may be replaceable by a micro-batch today — and it is 8× the cost (see the 💸 above), so the re-ask is worth putting on a calendar.

And be careful reading older material. A 2018 article arguing that streaming is necessary for a given latency was probably right in 2018. The concepts — event time, watermarks, state — are durable; the threshold is not.

🎓 Interview Angle — "we need this in real time"

Not an interview question — a question you will be asked at work, and the way you answer it is what the interview version is checking.

The weak response is to agree or to refuse. Agreeing builds an 8× cost for latency nobody uses; refusing makes you the person who says no.

The strong response converts it into a decision:

"Help me with one thing — what decision gets made on this, and what happens if it arrives thirty minutes later? If someone's looking at it in the morning, we can have fifteen-minute freshness on the batch stack next sprint. If a system is acting on it automatically at the moment of the event, that's genuinely streaming and it's a bigger piece of work — about eight times the compute and a quarter of an engineer ongoing — and I'd want to scope it properly."

Four things that response does. It asks about the decision rather than the latency. It offers something concrete and soon for the common case. It names the cost of the expensive option in both currencies. And it does not say no.

In an interview, the follow-ups are:

"They insist. Now what?" — build the fifteen-minute version, instrument the actual latency requirement, and revisit with evidence. "I'd escalate" is a weaker answer than "I'd ship something and measure."

"How do you know thirty minutes is the right question?" — because it is far enough beyond any batch cadence to be a real distinction and close enough that the answer is rarely "we'd lose money." It separates the four clusters in one question.

And "when have you built a streaming pipeline?" — where the honest answer for many candidates is "once, and we should not have." That answer, told well, is stronger than a longer list.

🏭 From the Pipeline — the streaming job that was replaced by a cron entry

A streaming job aggregated inventory movements into a per-SKU on-hand count, updated continuously, feeding an operations dashboard. It had run for two years and cost about $7,000 a month.

During a cost review somebody asked §29.1's question: what decision is made on this, and what does being late cost?

The answer took a week to get and it was: a reallocation between warehouses, decided twice a day, by a person, at 09:00 and 15:00. The dashboard was open continuously because it was on a wall.

The replacement was a fifteen-minute micro-batch, costing about $310 a month, and nobody noticed the change — which was confirmed by not announcing it for three weeks.

Three things about how it had happened.

Nobody had ever asked. The original requirement was "we need real-time inventory", from operations, and it was implemented as stated. The person who said it meant "sooner than the overnight batch."

The wall display made it look continuous. A dashboard refreshing every thirty seconds looks like it needs a streaming pipeline behind it, and it is the same dashboard whether the data is thirty seconds or fifteen minutes old.

And the job was working perfectly, which is why no review had ever looked at it. Cost reviews look at things that are broken or expensive; this was neither, at $7,000 a month, in a bill where compute was $18,873.

The generalisable finding: a streaming job's cost is invisible in the same way idle warehouse time is (Chapter 33 §33.7) — it is a steady, unremarkable line that has always been there. The thing that surfaces it is not a monitor; it is somebody asking what decision the data supports, and that question has no automated version.

29.12 The Kestrel Platform

🧱 Kestrel Platform — Increment 29: one streaming path, deliberately

Kestrel has exactly one genuine streaming requirement, and the increment's main work is establishing that and writing down why the others are not.

text platform/streaming/ inventory_alerts/ ← THE streaming job: stock-out detection job.py Flink; tumbling 1-min windows by SKU state.py explicit UIDs on every operator watermark.md the measured lag distribution and the choice ADR-014-why-not-streaming.md ← the four requests that were declined

The one that qualifies: a SKU going out of stock while the site still shows it available. The decision is made by code, per event, and being 30 minutes late means orders taken for stock that does not exist — which is a customer-visible error and a refund.

Six things this increment must get right:

1. The latency requirement is stated as a decision, not a number: "suppress the buy button within 60 seconds of stock reaching zero." §29.1.
2. The watermark comes from a measured distribution, in watermark.md, with the p99.9 and the max — and the late-data policy is one of §29.4's three, chosen and named.
3. with_idleness is set, because the topic is partitioned by SKU and most SKUs are quiet. §29.5.
4. The sink is idempotent — a merge on (sku, minute) — rather than transactional, because it is simpler and Chapter 20 §20.3 already covers it.
5. Explicit UIDs on every stateful operator, from version one, and a state TTL on the dedup set.
6. A freshness assertion on the SINK, not just consumer lag — because §29.5's idle-partition stall shows nothing on the job's own metrics.

And ADR-014 records the four declined requests, each with §29.1's question and its answer. That document is the increment's most valuable artifact, because it is what stops the same conversation happening four more times.

The exercise that matters is 29.23(b): measure your own processing_time − event_time distribution before choosing a watermark. Everyone picks 30 seconds. Almost nobody measures.

29.13 Summary

"Real time" is a shorthand for "sooner than now." The question that resolves most requests: 📐 "if this arrived thirty minutes later, what would go wrong?" Three of the four answer clusters do not need streaming, and the step from hourly to continuous is where the cost is — $23,000 to $61,000 a year, six extra weeks, and a new failure class on a four-person rotation.

Batch and streaming are the ends of a continuum. Micro-batch is under-used — 95% of the latency benefit at 20% of the complexity, with Chapter 20's machinery unchanged.

🔎 Lambda's fatal flaw was maintaining the logic twice and having the two paths disagree; Kappa's was assuming the log holds everything. Neither won: stream processors got exactly-once and event-time semantics, table formats got ACID, and micro-batch got good enough. The modern answer is one code path at whatever cadence the requirement needs. The surviving piece of Lambda — a fast approximate answer plus a slow correct one — is a useful technique, and the mistake was making it an architecture.

Event time is when it happened; processing time is when you saw it. ⚠️ A watermark forces three questions: how long to wait · what happens to late data · what downstream consumers do with a correction — the last being a contract question, not a streaming one. The default in most frameworks is to drop late events silently, which answers all three in the worst way.

Measure the lag distribution and choose from the tail. Kestrel's clickstream: p99 4.1 s, p99.9 28 s, max 16 minutes.

⚠️ An idle partition holds back the whole watermark — because a watermark is the minimum across partitions — and the job looks completely healthy while emitting nothing. with_idleness is one line. The only check that catches it is a freshness assertion on the sink.

"Exactly-once delivery" is impossible; "exactly-once processing" means the effects appear once. Achieved by a transactional sink or 🔁 an idempotent one — which is Chapter 20 §20.3, unchanged, and is usually simpler. The guarantee ends at your sink's boundary: external API calls, non- participating sinks, and anything downstream sit outside it.

State is where the operational difficulty is. How much · where it lives · how it is checkpointed · ⚠️ and when it expires — a deduplication set without a TTL dies in three months with an unattributable OOM.

📐 A join window is a business decision, not a tuning parameter. A watermark's delay describes your infrastructure; a join window describes what the business permits. Kestrel's payment lag was p99.9 of 42 seconds and a business answer of three days — which established that it is not a streaming join at all. When the business window and the affordable state window differ by orders of magnitude, you need both paths.

🧪 Streaming is deterministically testable, because event time is data. advance_watermark_to is the whole technique. Five cases, of which the last three are almost never written: out-of-order arrival · a late event inside allowed lateness · one outside it, asserting the drop is COUNTED · and a restart mid-window, which catches missing operator UIDs in a test rather than in a deploy.

Four failure modes batch does not have: consumer lag (alert on the ratio, not a threshold) · backpressure · checkpoint age · and a restart that resumes mid-window with state.

🏭 Deploying a stateful streaming job has no batch equivalent. Savepoint, restore, and the new job must read the old state — so set explicit operator UIDs from version one, before you need them. Chapter 27's deploy shape gains a fourth value: does this change the state schema?

Streaming is right when a decision is made per event by code, when volume makes batch impractical, when the source has no batch alternative (which argues for streaming ingestion, not streaming transformation), or when value decays in seconds. It is not right because the data arrives continuously, or because it would be elegant.

Chapter 30 turns from moving data to knowing about it: catalogs, lineage, access, and the question of where a number came from.

Key terms: event time · processing time · temporal join · interval join · watermark · allowed lateness · tumbling · sliding · session window · idleness · Lambda · Kappa · at-most-once · at-least-once · exactly-once processing · idempotent sink · transactional sink · state TTL · checkpoint · savepoint · operator UID · consumer lag · backpressure