Case Study 2: Four Hundred and Twelve Alerts

"A hundred and forty-seven of them nobody even clicked. Two of those were real."

Executive Summary

Kestrel's first alert review covered 412 alerts over three months. The categorization was uncomfortable:

Eighty-nine percent produced no action. And the 147 nobody clicked is the number that mattered, because two of them were real — a source that had stopped loading, and a grain violation — both of which were resolved weeks later by other means, with nobody connecting the resolution to the alert that had said so at the time.

The review cut the alert set to 61, of which 44 were acted on in the following quarter. The other 351 did not disappear: they became dashboard rows, which is where a measurement belongs when nobody is supposed to do anything about it right now.

Skills applied: alert design (§25.8); the test-versus-metric distinction (§25.2); routing (§25.8); rate limiting (Chapter 24 Case Study 2).

Background

How the 412 accumulated. Nobody decided to have four hundred alerts. Each one was added by somebody responding correctly to an incident, and the review's most useful finding is that the process that produced them was working exactly as designed.

"we didn't know the source was late"            → alert on source freshness
"the sensor sat there for an hour"              → alert on task duration
"nobody noticed the warehouse bill doubled"     → alert on daily spend
"we didn't know the scheduler restarted"        → alert on container restarts

Every one of those is a reasonable postmortem action item, and every one produced a notification into #data-alerts, and after two years there were 412 a quarter into a channel with six people in it.

Roughly four and a half a day, which is under the threshold at which anyone declares a problem and above the threshold at which anyone reads them all.

The Problem

The review was triggered by an incident that was not, at first, about alerting.

A supplier feed had stopped loading. It was found nine days later, by an analyst, and the postmortem's first question was the routine one: did we have an alert for this?

Yes. It fired on day one, and every day after.

#data-alerts   2026-06-03 04:12   [WARN] source freshness: supplier_inventory
#data-alerts   2026-06-04 04:12   [WARN] source freshness: supplier_inventory
   ⋮                              …nine of these…

Nine correct alerts, into a channel a human was in, none acknowledged.

⚠️ Failure Mode — "we need an alert for this" is the wrong postmortem action item

It is the most common action item in data postmortems and it is right often enough to be a reflex.

The problem is that it has no counterweight. Every incident adds one; nothing ever removes one; and the removal is the part that requires a decision nobody is incentivized to make. An alert set is a ratchet.

What makes it worse is that each addition is locally correct. The freshness alert on supplier_inventory was the right response to a real gap, it was well written, and it did exactly what it was built to do. It failed because of the 411 others, which is a property no reviewer of that one pull request could see.

The action item that has a counterweight:

"Add an alert for X, and name one existing alert to remove or downgrade, or state why the total should grow."

It is deliberately annoying. The friction is the point: it makes the aggregate visible at the moment of the decision, which is the same move as Case Study 1's margin budget in a pull request, and the only moment at which it is cheap.

And it produces a better conversation than a cap would. A hard limit on alert count invites gaming; a requirement to name what this is more important than invites a comparison, and the comparison is the thing you actually want made.

The Analysis

The categorization, done by hand over two days from the notification history and the incident log:

412 alerts, 2026-04-01 → 2026-06-30
├── 47   acted on                          11%
├── 218  acknowledged, no action           53%
└── 147  never acknowledged                36%

Step 1: what were the 218?

Almost all were one of three kinds, and none of the three is an alert:

A number moved and that was fine. Daily spend up 14%; row count down 8%; a job 40 seconds slower. These are §25.2's metrics wearing an alert's clothing — worth seeing, not worth interrupting for.

A known, tolerated condition. A source that is reliably late on Mondays. The alert was correct every Monday for two years.

A duplicate of something else that fired at the same time. A task failure plus its DAG failure plus its downstream skip: three notifications, one event.

Step 2: what were the 147?

This required going back through the incident log and asking, for each, whether anything had happened that the alert was about.

147 never acknowledged
├── 139  the same three kinds as above, from noisier sources
├── 6    genuinely ambiguous — no way to tell retrospectively
└── 2    REAL, and resolved later by other means

The two.

supplier_inventory freshness, which is the incident that triggered the review — nine alerts, nine days, found by an analyst.

A grain violation on silver.sessions, which had fired daily for 47 days and was found during Chapter 23's mute audit. The alert had been correct every single day.

🔎 Read the Plan — measure acknowledgment, not delivery

Every alerting tool reports delivery. Almost none report acknowledgment, and the gap between those two numbers is the only honest measure of whether an alerting system works.

text delivered 412 ← what the tool reports acknowledged 265 ← what a human touched acted on 47 ← what changed anything

The first number is the one on every vendor dashboard and it means nothing.

Three things to instrument, and all three are cheap:

• Acknowledgment rate per alert rule. A rule under 20% is being ignored, and it is being ignored whether or not it is correct.
• Time to acknowledgment. A rule whose median is four hours is not a page; reclassify it.
• Action rate. Requires an incident log, and it is the number that separates §25.8's "actionable" from "informative."

The uncomfortable implication: a rule with a 0% acknowledgment rate provides exactly the assurance of a rule that does not exist, even when every one of its firings is correct. That is Chapter 23's "measure whether a control is operating, not whether it exists," arriving from a fourth direction.

Step 3: why the two real ones were missed. Neither was a wording problem. Both were correct, specific, and named the table.

They were missed because they were indistinguishable from their neighbours in a channel where 89% of messages required nothing. A human scanning that channel is running a classifier trained on the base rate, and the base rate said ignore.

The Decision

Four changes.

One: reclassify by the §25.8 test. For each of the 412, one question: is there something a person should do, now?

The 274 are the important number. They were not wrong; they were the wrong instrument, and §25.2's distinction is exactly this — the test protects the pipeline, the metric informs a person, and an alert interrupts one.

Two: routing by who fixes it. Chapter 24 §24.12 and Chapter 23 §23.12:

#data-platform    scheduler, metadata DB, workers, disk       → platform
#data-alerts      DAG and task failures, test failures        → the DAG's owner
#data-sources     source freshness, schema drift              → the PRODUCING team
weekly digest     cost, duration trends, coverage             → nobody, in a meeting

#data-sources was the change with the largest effect, and it is the one that required the most negotiation: a source freshness failure is not the data team's to fix, and routing it to them had been producing an alert whose only possible action was to forward it.

Three: rate limiting. Chapter 24 Case Study 2's lesson. Any rule firing more than three times in an hour collapses into one message with a count.

Four: the quarterly review, with §25.8's two questions. Did anyone act? Should someone have?

📐 Design Decision — deleting an alert that has never been wrong

Sixty-three alerts were deleted as duplicates. A further nine were deleted despite having fired correctly, and that argument took most of an afternoon.

The example everyone argued about: a rule alerting when the nightly DAG's duration exceeded 4 hours. It had fired eleven times in two years and had been correct all eleven.

The case for keeping it: it is correct, it is cheap, and it has caught real problems.

The case for deleting it, which won: §25.4's duration-ratio rule catches everything this rule catches and more, earlier, and the absolute threshold at four hours had by then been passed routinely as the DAG grew — so it was firing on the normal state and its eleven correct firings were increasingly accidental.

The general principle: an alert is not justified by having been right. It is justified by being the best available instrument for a decision. A rule that is correct and redundant costs attention, and attention is the budget §25.8 is spending.

The counter-case, recorded honestly: the ratio rule needs fourteen runs of history and produces nothing on a brand-new job. The absolute threshold covers exactly the case the good rule cannot, which is why three of the nine were reinstated a month later — for new jobs only, with an expiry date on each.

What Happened

Forty-four actions from 61 alerts, against 47 from 412. Almost the same amount of work got done, from one seventh of the interruptions — and the three never-acknowledged alerts in Q3 were all the same rule, which was reclassified in the next review.

Two findings from the review that were not about alerting:

Nine rules had no owner. Not "owned by the data team" — no field, no name, nothing. Assigning them found that four were monitoring systems the team no longer ran.

The busiest rule fired 63 times and was a duplicate of the second-busiest. Both were added after the same incident, eight months apart, by people who did not know the other existed. There was no inventory of alert rules, which is the same shape as Chapter 25 §25.12's models with no readers and Chapter 23's mutes: an artifact accumulating with no register.

And a lesson about the review itself. It took two engineer-days, most of it reconstructing whether anyone had acted, from a chat history and an incident log that were never designed to be joined. Instrumenting acknowledgment up front would have made the next review an hour, and that is now done — which means the second review, six months later, cost ninety minutes and produced the same class of finding.

Lessons

1. 89% of alerts produced no action, and 36% were never acknowledged. Both numbers are typical and neither is usually measured.

2. A rule with a 0% acknowledgment rate provides the assurance of a rule that does not exist, even when every firing is correct.

3. Measure acknowledgment and action, not delivery. Delivery is on every vendor dashboard and means nothing.

4. "We need an alert for this" is a ratchet. Every incident adds one; nothing removes one. Pair it: add an alert and name one to remove or downgrade, or state why the total should grow.

5. Each addition is locally correct, which is why no reviewer of the individual pull request can see the problem. The friction has to be at the moment of the decision.

6. The test is: is there something a person should do, now? 274 of 412 were the wrong instrument — real signals that belonged on a dashboard.

7. Route by who fixes it. Source freshness belongs to the producing team; routing it to the data team produces an alert whose only action is to forward it.

8. An alert is not justified by having been right. It is justified by being the best available instrument for a decision — and a correct, redundant rule costs attention.

9. But the crude rule covers what the good rule cannot. Three of nine deletions were reinstated for new jobs, where a ratio needs history it does not have.

10. The two real misses were not wording problems. They were indistinguishable from neighbours in a channel where the base rate said ignore.

11. There was no inventory of alert rules — the same shape as models with no readers and mutes with no audit. An artifact accumulating with no register.

12. Instrument acknowledgment up front. The first review cost two engineer-days of archaeology; the second cost ninety minutes.

Questions for Discussion

1. The paired action item ("add one, name one to remove") is deliberately annoying. Where is the line between useful friction and a process people route around?

2. 274 alerts became dashboard rows. What makes a dashboard row get read, given that nothing interrupts anyone to read it?

3. Nine rules fired correctly and were deleted. Construct the strongest argument that this was a mistake, and say what evidence would settle it.

4. #data-sources routes freshness failures to the producing team, which required negotiation. What does that team need in order to accept it, and what does Chapter 17 provide?

5. The two missed alerts were correct and specific. If wording was not the problem, what would you change so a correct alert in a noisy channel gets seen?

6. Both this case study and Case Study 1 turn on a register that did not exist — alert rules, and the margin. How many registers should a platform have before the registers themselves become the problem?

7. Estimate your own numbers: how many alerts fired last month, how many were acknowledged, how many changed anything? If you cannot answer the second and third, what would it take?