Case Study 2: The Real-Time Requirement That Was a Scheduling Bug

"They had budgeted six weeks and a Flink cluster. The fix was changing a cron expression, and it took eleven minutes."

Executive Summary

Kestrel's merchandising team escalated a request for real-time inventory and sales data. The stated need was urgent, the business case was credible, and an initial estimate came back at six weeks of engineering and about $38,000 a year of additional infrastructure.

The request was granted a design review before the build, which is the only reason this is a short case study.

The review asked §29.1's question — "if this arrived thirty minutes later, what would go wrong?" — and the answer, after twenty minutes, was nothing. The merchandisers make repricing decisions at 09:00 and 14:00. They did not need real-time data; they needed the 09:00 data to exist by 09:00, and it did not, because agg_product_daily ran at 10:40.

It ran at 10:40 because of a cron offset written in 2024 to give an extract "enough time," on a schedule that had since moved twice.

The fix was one line. The data was available at 06:20 the next morning, and has been since.

Skills applied: §29.1's latency question; the spectrum (§29.2); and the general observation that a technology request is frequently a symptom of an unexamined operational fact.

Background

The request, as submitted:

"We need real-time visibility of inventory and sales by product. Currently the data is hours old by the time we see it, which means we are repricing on stale information and losing margin. Competitors have this."

Every sentence is true and the conclusion does not follow, which is the general shape of these requests and is why they are hard to decline well.

The team's initial estimate, done properly:

Kafka topics for inventory + order events        already exist (Ch. 15)
Flink job: per-product aggregation                3 weeks
serving layer: a low-latency store                2 weeks
operations: runbooks, monitoring, on-call         1 week
                                                  ────────
                                                  6 weeks

infrastructure                                  $38,000/yr
on-call                                    a new failure class

The estimate was accepted as accurate. The design review was not about whether it could be built.

The Problem

The review's first question, per §29.1: what decision is made on this data, how often, and what does being late cost?

Q: When do you reprice?
A: Twice a day. Morning meeting at 09:00, and again around 14:00.

Q: What data do you look at in the 09:00 meeting?
A: The product performance dashboard.

Q: What is on it at 09:00?
A: Yesterday's numbers. That's the problem.

Q: What would you want to be on it?
A: Yesterday's numbers.

Q: ...
A: I mean I want yesterday's numbers to be THERE. Right now it still shows
   the day before, until about eleven.

The requirement was not real-time. It was "yesterday's data, before the 09:00 meeting" — a twelve-hour latency requirement, which is the leftmost column of §29.2's spectrum and is what the platform already did everywhere else.

Why it was not met:

03:00  kestrel_daily starts
03:47  fct_order_item complete
03:52  gold.daily_revenue complete       ← the 6am SLA's target, met daily
       ...
10:40  agg_product_daily                 ← a SEPARATE DAG, on its own cron
10:52  the product dashboard refreshes

agg_product_daily was not part of kestrel_daily. It was written in 2024 as a standalone job, scheduled at 10:40, and 10:40 was chosen as "safely after the extract finished" — when the extract ran at 09:00 and took about an hour.

The extract has since moved twice, most recently to 03:00 as part of the work in Chapter 24. The downstream job's cron was never revisited, because nothing connected them.

⚠️ Failure Mode — a cron offset is a dependency that nothing maintains

This is Chapter 24 §24.1's argument arriving as an incident rather than an illustration.

10:40 encoded a dependency: "after the extract." That dependency was correct when written and became wrong the moment the extract moved — and nothing anywhere connected the two facts.

• No test failed, because both jobs succeeded.
• No alert fired, because there was no SLA on this dashboard.
• The person who moved the extract had no way to know, because the coupling existed only in a number in a different repository.
• The merchandisers experienced it as "the data is slow", which is a true statement about a symptom and gave nobody a reason to look at a cron expression.

The general form: an offset between two schedules is a dependency written in a form that cannot be checked, and it decays silently. Chapter 24 §24.7's ExternalTaskSensor execution_delta is the same hazard one layer up, and datasets are the fix for both.

The audit that finds these is one query against your scheduler, and it is worth running:

sql -- Every scheduled job, and the gap between it and the thing it depends on. -- Any gap you cannot explain in one sentence is a decayed dependency.

Kestrel found three more, one of which was a job scheduled 90 minutes after a source that had been decommissioned entirely — it had been reading a table nothing wrote to for five months, and producing an output nobody had noticed was frozen.

The Analysis

Step 1: could agg_product_daily simply move?

Yes. It depends on fct_order_item, which completes at 03:47.

Step 2: why was it separate at all? History. It was written by a different person, for a different stakeholder, before kestrel_daily existed in its current form. No technical reason survived examination.

Step 3: what would real-time actually have bought?

The review did this calculation deliberately, because "you do not need it" is not a satisfying answer without one:

decision cadence                  twice daily (09:00, 14:00)
data age at decision, before      27 hours (yesterday, at 11:00 today)
data age at decision, after fix   9 hours  (yesterday, at 06:20 today)
data age at decision, streaming   ~2 minutes

value of going from 27h to 9h     the entire complaint
value of going from 9h to 2 min   nothing -- the meeting is at 09:00

The second row is the whole finding. A decision made at a fixed time cannot use data fresher than the last time it was refreshed before that moment — so the useful latency is bounded by the decision cadence, not by the technology.

📐 Design Decision — how to decline a request without declining the person

The review had to say no to something a team had escalated, with a real business complaint behind it. Three things made it land, and the order matters:

Fix the actual problem first, then discuss the request. The cron change shipped the same afternoon; the conversation about streaming happened the following week, with the merchandisers already having what they had asked for. A "no" delivered alongside a fix is a different conversation from a "no" delivered instead of one.

Show the arithmetic, not the conclusion. The three-row table above was the entire argument, and the merchandisers reached the conclusion themselves — one of them pointed out the 14:00 meeting before anyone else did.

Name the case where they would be right. The review stated explicitly: "if you ever need to reprice automatically, in response to a stock level, without a person in the loop — that is a genuine streaming requirement and we will build it." This matters more than it sounds: a stakeholder told "no" hears "not for you"; a stakeholder told "not for this, and here is what would change it" hears a criterion they can apply themselves.

What the review did NOT do, and deliberately: it did not say the request was wrong, or naive, or that "real-time" is a buzzword. The complaint was correct, urgent, and costing margin. Only the proposed solution was wrong, and separating those two is most of the skill.

The outcome nine months on: the merchandising team has raised two further data requests, both stated as "we make this decision at this cadence and need the data by then" — which is the vocabulary from the review, coming back.

The Decision

Three changes, and the third is the one that generalizes.

One: move the job. agg_product_daily became a task in kestrel_daily, downstream of fct_order_item via a ref() (Chapter 19 §19.3) rather than a cron offset. Eleven minutes, including the pull request.

Two: an SLA on the product dashboard. It had never had one, which is why nothing detected a four-hour regression when the extract moved. 08:00, and it is now in the Chapter 26 §26.2 SLI set.

Three: a standing design review for any request framed as a technology.

🏭 From the Pipeline — the four questions that go in front of every "we need X" request

Kestrel's review is four questions and takes twenty minutes. It has been run eleven times in a year and has resulted in the requested technology twice.

1. What decision is made on this, by whom, and how often? If the answer is "nobody decides anything, we look at it" — the requirement is a dashboard, not an architecture.
2. If it arrived thirty minutes later, what would go wrong? §29.1. Three of the four answer clusters do not need the technology.
3. Is the current thing meeting its own stated requirement? ← this one found the bug. A request for something faster is often a request for the existing thing to work.
4. What would we build if the answer to 3 is yes? Asked last, deliberately, so the design conversation happens after the diagnosis rather than instead of it.

Question 3 is the one most often skipped, because it sounds like deflection — "have you tried turning it off and on again" addressed to a stakeholder. Asking it well means asking it about your own system, not about their request, and phrasing it as "before we design something, let me check whether what we have is doing what we told you it does" has never once been received badly.

Of the eleven reviews: two produced the requested build, six produced a fix to something existing, and three produced a different, smaller thing. The six are the interesting number.

What Happened

The six-week build was not needed and the complaint was real. Both things.

The audit of scheduled jobs found three more decayed cron offsets:

A job 90 minutes after a decommissioned source. It had been reading a table nothing wrote to for five months and producing a frozen output. Nobody had noticed, which means nobody was reading its output either — Chapter 25 §25.12's unread-models problem, arriving from the other direction.

A weekly report scheduled 30 minutes after a monthly job, which meant it was correct on the first week of the month and stale for the other three. It had been like that since 2024.

A job whose offset was still correct — and which nobody could explain, because the person who set it had left. It was left alone, with a comment saying exactly that, which is the honest outcome for one of the four.

And the review's vocabulary spread. Three subsequent requests from other teams arrived already phrased as a decision and a cadence, which the team attributes to the merchandisers having explained the review to their peers rather than to anything the data team did.

Lessons

1. "We need real-time" is almost never a latency requirement. It is a complaint about a symptom, and the complaint is usually correct.

2. §29.1's question resolves most of them in twenty minutes: if this arrived thirty minutes later, what would go wrong?

3. A decision made at a fixed time cannot use data fresher than the last refresh before it. The useful latency is bounded by the decision cadence, not by the technology.

4. ⚠️ A cron offset is a dependency written in a form nothing can check, and it decays silently when the thing it depends on moves. No test fails, no alert fires, and the person who moved the upstream had no way to know.

5. Ask question 3 — is the current thing meeting its stated requirement? It is the most-skipped question and it found this bug. Ask it about your own system, not about their request.

6. Fix the actual problem before discussing the request. A "no" delivered alongside a fix is a different conversation.

7. Show the arithmetic and let them reach the conclusion. One of the merchandisers raised the 14:00 meeting before anyone else did.

8. Name the case where they would be right. "Not for this, and here is what would change it" gives a stakeholder a criterion they can apply themselves — and nine months later they were applying it.

9. The complaint was correct and the proposed solution was wrong. Separating those is most of the skill, and conflating them is how a data team acquires a reputation for saying no.

10. The audit found three more decayed offsets, including a job reading a decommissioned source for five months, and one that is still correct and unexplained — left alone, with a comment saying so.

11. Of eleven design reviews, six produced a fix to something existing and only two produced the requested build.

Questions for Discussion

1. The estimate of six weeks was accurate. Does producing an accurate estimate for the wrong thing count as good work?

2. Question 3 sounds like deflection when asked badly. Write the version you would actually say, for a stakeholder you know.

3. The review names the case where the requester would be right. What is the risk of that, and how would you avoid inviting a request engineered to meet the criterion?

4. A cron offset encodes a dependency nothing can check. What would a scheduler have to expose for that audit to be automatic?

5. One decayed offset was still correct and unexplained, and was left alone with a comment. Is that the right call? What would change it?

6. The job reading a decommissioned source ran for five months producing frozen output that nobody noticed. What does that tell you about the value of its output, and what should happen to it?

7. Six of eleven reviews found a problem with an existing system rather than a need for a new one. What does that ratio suggest about where a data team's improvement effort should go?