Case Study 1: Three Weeks of Nothing, Every Night

"Consumer lag: zero. Checkpoints: succeeding. Backpressure: none. CPU: fine. And between 03:40 and 06:20 the job produced no output at all."

Executive Summary

Kestrel's stock-out detection job — the platform's one genuine streaming workload (§29.10) — stopped emitting between roughly 03:40 and 06:20 every night for three weeks.

Every metric the job exposes was healthy throughout. Consumer lag was zero, checkpoints succeeded on schedule, there was no backpressure, and CPU sat at 12%.

The topic is partitioned by SKU. Overnight, traffic falls enough that some partitions receive no events for several minutes — and a watermark across parallel partitions is the minimum of the per-partition watermarks. One quiet partition held the global watermark back, and no window could close.

The cost was eleven stock-outs not detected during the overnight window, of which four resulted in orders taken for stock that did not exist, and $4,207 in refunds and expedited replacements.

The fix is one line: with_idleness(Duration.of_minutes(1)).

Skills applied: per-partition watermarks (§29.5); absence-shaped failures (Chapter 25 §25.9); freshness assertions on the sink; and the deterministic test that would have caught it (§29.9).

Background

The job. inventory_alerts consumes a stream of stock-level changes and emits an alert when a SKU's available quantity reaches zero, so the storefront can suppress the buy button.

The requirement, stated as a decision (§29.1): "suppress the buy button within 60 seconds of stock reaching zero."

The topology:

env.add_source(kafka_source)
   .assign_timestamps_and_watermarks(
        WatermarkStrategy.for_bounded_out_of_orderness(Duration.of_seconds(30)))
   .key_by(lambda e: e.sku)
   .window(TumblingEventTimeWindows.of(Time.minutes(1)))
   .aggregate(MinAvailableQuantity())
   .add_sink(alert_sink)

Twelve Kafka partitions, keyed by SKU. 4,120 active SKUs, and the distribution of stock changes across them is extremely uneven — a handful of fast-moving products account for most events.

During the day this is invisible. Every partition sees traffic within any given minute.

Overnight it is not. Between 03:00 and 06:00 Eastern, Kestrel's order rate is at its daily minimum, and a partition holding only slow-moving SKUs can go five or ten minutes without an event.

The Problem

The gap was found by accident. An analyst building a report on stock-out frequency noticed that the alert table had no rows at all in a consistent nightly window:

SELECT date_trunc('hour', alerted_at) AS h, COUNT(*)
  FROM gold.stock_out_alerts
 WHERE alerted_at >= current_date - 21
 GROUP BY 1 ORDER BY 1;

hour (ET)   alerts   (21-day totals)
   01:00        38
   02:00        31
   03:00        22
   04:00         0     ←
   05:00         0     ←
   06:00        11
   07:00        47

Two hours with exactly zero, every night, for twenty-one nights.

The first hypothesis was that stock does not run out at 04:00, which is superficially plausible — overnight order volume is low. It was tested and rejected in one query:

-- Stock levels DID reach zero overnight. The events exist upstream.
SELECT COUNT(*) FROM silver.stock_changes
 WHERE available_qty = 0
   AND changed_at::time BETWEEN '04:00' AND '06:00'
   AND changed_at >= current_date - 21;
-- 11

Eleven stock-outs happened. Zero alerts were emitted.

⚠️ Failure Mode — every metric the job emits describes the job, and the job was fine

This is the shape that made three weeks possible, and it is Chapter 25 §25.9's absence problem in a place where it is especially well hidden.

What was monitored, and what each measures:

Metric	Reading	What it actually measures
consumer lag	0	records consumed, not records emitted
checkpoint age	41 s	the job's state is being saved
backpressure	none	no stage is slower than its upstream
CPU / memory	12% / 38%	the job is not working hard
records-in rate	normal	events are arriving
records-out rate	0	← not monitored

The job was consuming, checkpointing, and doing nothing with what it consumed, and only the last row would have said so.

Consumer lag being zero is the actively misleading one. It reads as the healthiest possible signal and it means "we have read everything available" — which is exactly what a job that reads and buffers looks like. A stalled watermark produces perfect consumer lag, because consumption is not the thing that stopped.

The general rule for a streaming job, and it applies to every pipeline in this book: monitor the OUTPUT, not the machinery. Chapter 25 §25.2's freshness signal, on the sink:

sql -- The one check that catches this. Everything else was green. SELECT max(alerted_at) FROM gold.stock_out_alerts; -- expect < 15 min old

It is four lines and it was the last thing anyone thought to add, because a streaming job comes with a large number of built-in metrics and having many metrics feels like having monitoring.

The Analysis

Step 1: is the watermark advancing? The Flink UI exposes it per operator, and the answer was immediate once someone looked:

03:38   watermark  2026-09-14 03:37:28
03:44   watermark  2026-09-14 03:37:28     ← unchanged
03:51   watermark  2026-09-14 03:37:28
04:12   watermark  2026-09-14 03:37:28

The watermark stopped at 03:37:28 and did not move for over two hours, while records continued to arrive.

Step 2: why? The per-partition watermarks told the whole story:

partition   last event      partition watermark
    0       04:11:52        04:11:22
    1       04:11:47        04:11:17
    2       04:11:50        04:11:20
    ...
    7       03:37:58        03:37:28    ← nothing since 03:37
    ...
   11       04:11:44        04:11:14

global watermark = MIN(all) = 03:37:28

Partition 7 holds slow-moving SKUs. It had not received an event in 34 minutes, and because a partition with no events has no basis for advancing its watermark, it pinned the global minimum.

Step 3: why does the daytime work? Because every partition sees an event within any given minute, so the minimum advances continuously. The bug is present all day and has no effect all day, which is why it survived the job's initial testing and its first four months in production.

Step 4: what did it cost?

overnight stock-outs not alerted, 21 nights      11
  → of which the storefront kept selling          4
  → orders taken for unavailable stock           17
refunds + expedited replacement                 $4,207

The other seven were caught by the 06:20 recovery — once traffic picked up, partition 7 received an event, the watermark jumped forward by two and a half hours, and every held window closed at once.

🔎 Read the Plan — the recovery is what made it survivable, and it is also what hid it

When the watermark finally advanced at 06:20, every buffered window emitted, in one burst.

So the alerts were not lost. They were late — by up to two and a half hours — and they arrived with correct event-time timestamps, which meant:

• The alert table looked almost right. Rows existed for the overnight period; the analyst's query grouped by alerted_at (processing time) and found the gap, and a query grouping by event_ts would have shown nothing wrong at all.
• Any daily aggregate was correct. The day's totals reconciled perfectly.
• Only the operational use was broken, which is the one use the job exists for.

This is worth generalizing, because it is a trap in any event-time system: correctness and timeliness are separate properties, and an event-time pipeline can be perfectly correct and entirely useless.

A reconciliation cannot detect it. A daily total, a row count, a sum — all correct. The only thing that detects it is a measurement of the gap between event time and emission time:

sql SELECT max(alerted_at - event_ts) AS worst_lag FROM gold.stock_out_alerts WHERE alerted_at >= current_date - 1;

That single column is the SLI for a streaming job (Chapter 26 §26.2), and it is the one Kestrel did not have — because their SLIs were written for batch, where event time and emission time differ by the schedule and the schedule is known.

The Decision

Four changes.

One: with_idleness.

WatermarkStrategy
    .for_bounded_out_of_orderness(Duration.of_seconds(30))
    .with_idleness(Duration.of_minutes(1))     # ← the fix

One minute, chosen because it is well above the 30-second watermark delay and well below the 60-second requirement. A partition silent for a minute stops holding the watermark back.

Two: a freshness assertion on the sink. The four-line query from the ⚠️ callout, every five minutes, alerting above fifteen minutes.

Three: an end-to-end latency SLI. emission_time − event_time, published per §25.4's five-column table and alerted on the ratio to a trailing median.

Four: the test. §29.9's harness, with the case that reproduces it:

h = Harness(window_seconds=60, watermark_delay=30, partitions=2,
            idle_timeout=None)
for t in (0, 30, 90, 150):
    h.process(Event("k0", t, partition=0))     # partition 1: silent
h.advance_processing_time(300)
assert h.output() == []          # ← the bug, asserted

📐 Design Decision — with_idleness is not free, and the trade should be stated

The fix looks like a pure win and it is not. Declaring a partition idle means proceeding without it, and if that partition then produces an event with an earlier timestamp, that event is late — possibly outside allowed lateness, in which case it is dropped.

So with_idleness trades a stall for a small probability of dropped data, and the trade is right here for a specific reason worth stating: this job's output is an operational alert with a 60-second requirement, and an alert two hours late is worth less than no alert. A held window is not "safe"; it is a different failure.

Where the trade would go the other way: a job computing a financial aggregate, where a stalled window is recoverable (it emits eventually, correct) and a dropped event is not. For that job the stall is preferable, and the right response is to alert on watermark staleness rather than to set idleness.

The general form: with_idleness is correct when timeliness matters more than completeness, and wrong when it does not. That is a business question, and it is the same question §29.4's late-data policy asks — so answer both at once, and write the answer next to the code.

Kestrel's comment reads:

```python

idleness=1min. This job alerts operationally within 60s, so a stalled

window is worse than a dropped late event. For any job where the

aggregate matters more than the latency, do NOT copy this -- alert on

watermark staleness instead. Ch. 29 §29.5.

```

What Happened

Three dropped events in the first month, all on partition 7, all with event times inside a window that had already closed. Each was counted, which is §29.9's fourth test case, and the count is reviewed monthly — because three is fine and three hundred would mean the idleness timeout is too short.

Two further findings.

The job had been in production for four months before the analyst's query. The gap had existed from the first night; the three weeks in the summary is the period the analyst's report covered, not the duration of the bug. Nobody had ever looked at the alert table's distribution over time.

And the daily reconciliation had been passing throughout, because the alerts eventually arrived with correct event timestamps. Kestrel's Chapter 23 §23.4 assertions — grain, volume floor, freshness, referential, distribution, business rule — all passed, and the volume floor passed because the alerts did arrive, just hours late.

Which produced a change to the assertion register (Chapter 23 §23.13): a twenty-third entry.

End-to-end latency, for anything with a timeliness requirement. max(emission_time − event_time) against the requirement. None of the other twenty-two measures it, because they were all written for batch, where the schedule makes latency a known constant.

Lessons

1. A watermark across parallel partitions is the MINIMUM. One silent partition holds everything back, and the job stops emitting while every metric it exposes stays healthy.

2. Consumer lag of zero is the actively misleading signal. It means "we have read everything available," which is exactly what a job that reads and buffers looks like.

3. Monitor the OUTPUT, not the machinery. A four-line freshness query on the sink was the only check that would have caught it, and it was the last thing anyone thought to add — because a streaming job comes with many built-in metrics and having many metrics feels like monitoring.

4. The bug is present all day and has no effect all day. It required a period of low traffic, which is why it survived testing and four months in production.

5. Correctness and timeliness are separate properties. The alerts were correct, with correct event timestamps, and arrived up to 2h 43m late — so every reconciliation passed and the job was useless for its only purpose.

6. The recovery burst is what hid it. Grouping by alerted_at found the gap; grouping by event_ts would have shown nothing wrong.

7. max(emission_time − event_time) is the SLI for a streaming job, and it is the twenty-third entry in the register — because the other twenty-two were written for batch, where the schedule makes latency a known constant.

8. with_idleness is not free. It trades a stall for a probability of dropped late data, and it is right only when timeliness matters more than completeness. Write that reasoning next to the code, including the instruction not to copy it.

9. Count the drops. Three a month is fine; three hundred means the timeout is too short.

10. Nobody had ever looked at the alert table's distribution over time. The finding came from an unrelated report, which is where four of this book's incidents have come from.

Questions for Discussion

1. Six metrics were monitored and none would have caught this. What is the general principle for choosing which metrics a streaming job needs, given that it emits dozens by default?

2. The job's correctness was never in doubt and it was useless. How would you write an SLO that captures that distinction (Chapter 26 §26.2)?

3. with_idleness trades a stall for dropped data. For a job you know, which way does the trade go — and is that written down anywhere?

4. The bug existed from the first night and was found four months later by an unrelated report. What would have found it in week one?

5. All twenty-two of Chapter 23's assertions passed. Is adding a twenty-third the right response, or does that register have a structural gap?

6. The recovery burst delivered correct data hours late. Are there consumers for whom that is fine? How would you know which of your consumers those are?

7. §29.9's test is five lines and reproduces a bug that ran for four months. Why do you think streaming jobs are so much less tested than batch ones?