Case Study 1: The Outage With a Date On It

"We have never missed the SLA. We are going to miss it on the second of December, and we know that because we finally plotted the right number."

Executive Summary

Kestrel's 6am SLA had been met every single day for two years. By the only measurement anyone had — a compliance rate — the platform was performing perfectly.

In August 2026 someone plotted the margin: minutes of headroom between the DAG finishing and the 06:00 deadline. The chart fell from 4h41m to 41 minutes, at 10.4 minutes a month, in a straight line.

$$\frac{41\ \text{minutes}}{10.4\ \text{minutes/month}} = 3.9\ \text{months}$$

The forecast named 2026-12-02. The team then had to do something no data platform team is practised at: argue for work on a failure that had not happened.

The chart's steps decomposed into six changes, and every one of them is an incident from an earlier chapter of this book. Fixing five recovered 168 minutes, taking the margin from 41 back to 3h29m and moving the date out by twenty months.

Skills applied: the margin as a metric (§25.5); duration ratios (§25.4); consumer monitoring (§25.12); and the retroactive lesson that a slow-moving aggregate is a sum of decisions.

Background

The SLA, from Chapter 1 §1.7: gold.daily_revenue is current for the prior day by 06:00 America/New_York, because the CEO opens the dashboard at 06:15.

How it was measured, for two years:

SELECT COUNT(*) FILTER (WHERE met) * 100.0 / COUNT(*) AS pct
  FROM sla_history WHERE d >= current_date - 90;
-- 100.0

One hundred percent, every quarter, for eight quarters. It appeared in a monthly report, it was the platform's headline number, and it was completely true.

What nobody plotted was how close each of those successes was.

The Problem

The margin was computed retroactively from two years of run_records — the five-column table from §25.4, which Kestrel had been writing since the platform's first month and had never queried:

margin to 06:00, monthly median
2024-09   4h 41m  ████████████████████████
2025-03   4h 12m  █████████████████████
2025-09   3h 26m  █████████████████
2026-03   2h 39m  █████████████
2026-07   1h 17m  ██████
2026-08   0h 41m  ███

$$281 - 41 = 240\ \text{minutes lost over 23 months} = 10.4\ \text{min/month}$$

Every month on that chart is a 100% compliance figure.

⚠️ Failure Mode — the metric was already being collected and had never been looked at

This is the part worth sitting with, because it is not a story about missing instrumentation.

run_records had finished_at on every run since 2024. The margin was one subtraction away, for two years, and it required no new collection, no new tool, and no budget.

What was missing was the question. The team measured compliance because compliance is what the SLA is stated as, and the shape of a measurement follows the shape of the promise — a promise phrased as a boolean produces a boolean.

Three other places the same pattern appears in this book, and it is worth checking your own for all three:

• A test's pass/fail hides how close it came (Chapter 23 §23.11 — coverage, not pass rate).
• A job's success/failure hides duration and memory (Chapter 21 CS2, Chapter 22 CS1).
• A reconciliation's within-tolerance hides the variance's trend (Chapter 20 CS2 — the 0.02% → 0.037% step change).

All four are the same move: a continuous quantity reported as a boolean, and the boolean is what gets stored.

The audit that finds these is one question asked of every green indicator you have: what number was thresholded to produce this? Then plot that number. It costs an afternoon and it needs no new data, because the number is already there — that is what makes it a threshold.

The Analysis

Step 1: is the trend real, or is it an artifact?

Three things were checked before anyone acted, and checking them is what made the forecast credible enough to fund:

Is it linear? Fitting a slope to the 700-run series gave −10.4 min/month with the residuals showing clear steps rather than noise — which is a better answer than a good fit, because steps are attributable.

Is the deadline itself stable? Yes. 06:00 America/New_York, unchanged, and the daylight-saving transitions appear in the chart as two 60-minute discontinuities that had to be handled — and handling them wrongly would have produced a slope that was entirely an artifact of March and November.

Is the start time stable? Mostly. The DAG starts at 03:00 and had drifted by four minutes over two years due to scheduler queueing. So the erosion is duration, not start time — which matters, because they have different fixes.

Step 2: decompose the steps. Each visible step in the chart was matched against the deploy log:

🔎 Read the Plan — a slow-moving aggregate is a sum of decisions

Look at that table again. Five of its six rows are incidents documented in earlier chapters of this book, each of which was found for an unrelated reason, and each of which was fixed as a local problem.

Nobody connected them, because there was nothing to connect them to. The sensors were a worker- slot problem. The snapshot was a modelling problem. The merge was a cost problem. The margin is the one number all five show up in, and it did not exist.

This is the strongest available argument for a single aggregate metric per platform. Not because the aggregate diagnoses anything — it does not; the decomposition took a day of matching steps against a deploy log — but because it is the thing that makes a set of unrelated small regressions visible as one problem with a size.

The test for whether you have one: can you name a number that would have moved for every incident in your last four postmortems? If not, the postmortems are describing independent events and some of them are not.

Step 3: the forecast, and its honesty.

$ python health.py run_records.json --margin-forecast

margin forecast: kestrel_daily
  current            41 min
  slope              -10.4 min/month over 700 runs
  reaches zero       2026-12-02  (117 days)

The team was explicit about what that date is not. It assumes the trend continues, and the trend is a sum of decisions that have not yet been made. A quiet quarter pushes it out; one bad deploy pulls it in.

What it is is a defensible answer to "when does this become a problem," which is the question every funding conversation actually turns on and which nobody had been able to answer.

The Decision

The hard part was not technical. Every fix in the table was known, and three had already been written up in postmortems and not prioritized, because each was individually small.

📐 Design Decision — funding work for a failure that has not happened

The proposal was two engineer-weeks to recover margin. The objection was reasonable and was made by a reasonable person: "we have never missed this SLA; why is this urgent?"

Three framings were tried. The first two failed and it is worth knowing why.

"The trend is bad." Failed. A trend is an argument about the future, and the future is where everyone's competing priority also lives.

"These five things are inefficient." Failed, and deserved to. Each was individually small and "make things faster" competes badly against work with a named beneficiary.

"On the second of December the CEO opens an empty dashboard." Worked, immediately.

The difference is not rhetoric. The third framing supplies a date and a named consequence, which are the two things a prioritization conversation runs on and which "the trend is bad" cannot supply. The forecast is what converted a quality argument into a scheduling argument, and the team's own retrospective note is worth quoting:

"We had been asking for time to make things better. What worked was asking for time to stop a specific thing on a specific day. Those are the same work."

And the honest caveat, which they recorded: a forecast used this way will be wrong, and being wrong in the safe direction — the date passes, nothing happens — costs credibility for the next one. They committed to re-publishing the forecast monthly, including when it moved out, so that the number is a measurement rather than an argument that was made once.

Five of the six were fixed, in this order — cheapest first, because the ordering itself made the case:

The sixth — data growth — was not fixed, because it is not a defect. It was budgeted instead: 41 minutes over 23 months is about 1.8 minutes a month of unavoidable erosion, which is now a line in the platform's capacity plan rather than a surprise.

What Happened

The compliance number did not change, because it could not. It was 100% before and after, which is the case study's whole point in one row.

Three things the team kept:

The margin on the dashboard, next to compliance, with the 90-day trend. §25.7.

A monthly forecast, published whether or not it moved. The commitment from the callout above.

A margin budget in the change process. Any change expected to add more than five minutes to the nightly DAG now states the number in its pull request. Not to block it — most such changes are worth it — but so that the erosion is a sequence of decisions rather than a discovery.

🏭 From the Pipeline — the change that costs 90 seconds, forty times

The margin budget found something the incident list did not.

No single change had ever been rejected on performance grounds, because no single change had ever been large enough to argue about. The sixteen sensors were sixteen separate pull requests, each adding three or four minutes, each obviously fine.

A 90-second addition is unarguable. Forty of them are an outage.

The budget makes the aggregate visible at the moment of the decision, which is the only moment at which it is cheap. The pull-request line is one sentence:

text Margin impact: +3m20s (measured on staging). Remaining margin 3h26m.

It has never blocked a change. It has caused four authors to look for a cheaper approach before opening the PR, which is the effect worth having and is not one a review gate produces.

Lessons

1. A pass/fail SLA cannot see erosion. 100% compliance for eight quarters, over a margin falling from 4h41m to 41 minutes.

2. The metric was already collected. finished_at had been in run_records since 2024, and the margin was one subtraction away. What was missing was the question.

3. The shape of a measurement follows the shape of the promise. A promise phrased as a boolean produces a boolean.

4. Ask of every green indicator: what number was thresholded to produce this? Then plot that number. It costs an afternoon and needs no new data — the number is already there, or there would be no threshold.

5. Check the trend is real before acting on it. Linearity, a stable deadline, a stable start time, and daylight-saving handled — getting that last one wrong would have produced a slope that was entirely an artifact.

6. A slow-moving aggregate is a sum of decisions. Five of the six steps were incidents from earlier chapters, each found for an unrelated reason and fixed as a local problem. The margin is the one number they all appear in.

7. The test for having such a number: would it have moved for every incident in your last four postmortems?

8. "The trend is bad" and "these things are inefficient" both fail as funding arguments. "On the second of December the CEO opens an empty dashboard" worked immediately — it supplies a date and a named consequence, which is what a prioritization conversation runs on.

9. Publish the forecast monthly, including when it moves out. A number used once as an argument stops being a measurement.

10. Budget the erosion you cannot fix. 1.8 min/month of data growth is a capacity-plan line, not a defect.

11. A 90-second addition is unarguable; forty of them are an outage. Put the margin impact in the pull request — it has never blocked a change and has caused four authors to find a cheaper approach.

Questions for Discussion

1. The team measured compliance because that is how the SLA was written. How would you phrase an SLA so that the natural measurement is continuous?

2. §"The Analysis" checked three things before trusting the trend, and the daylight-saving one would have invalidated it. What else could have produced a spurious slope here?

3. The funding argument that worked supplied a date. Is that manipulation, good communication, or both? What obligation does it create?

4. The forecast will eventually be wrong in the safe direction — the date passes and nothing happens. How should the team handle that, and what does it cost if they handle it badly?

5. Five of six steps were already-known incidents that had been deprioritized individually. Is there a process that would have connected them without the margin chart?

6. The margin budget has never blocked a change. Is that evidence it is working, or evidence it is theatre? What would distinguish the two?

7. Apply lesson 4 to your own platform: name three green indicators and the continuous number behind each. How many of those numbers are plotted anywhere?