Case Study 2: Muted in March

DataField.Dev

Case Study 2: Muted in March

"There was a thumbs-up emoji on the message. That was the whole change-management process."

Executive Summary

A freshness check on bronze.pricing_feed fired at 04:12 on a Saturday in March. The on-call engineer confirmed the feed was late because of a known supplier maintenance window, muted the alert in Slack with /mute pricing_feed_freshness 48h, and went back to bed. That was the correct action.

The mute did not expire. The Slack integration's 48h argument was accepted, logged, and — because of a defaulting bug in a wrapper the team had written — stored as an indefinite suppression.

The check ran and reported for 511 days without anyone seeing a result. During that time the pricing feed stalled twice more, for four days and for nine days, and dim_product.list_price_cents served stale prices into the margin report through both.

It was found by mute_audit.py, added in the §23.11 metrics work, on its first run.

Four mutes were found. Two were over a year old. One of them was the only check on its table.

Skills applied: severity and mutes (§23.7); data quality as a metric (§23.11); ownership (§23.12); the difference between a control existing and a control operating.

Background

The check, which is correct and well-written:

sources:
  - name: pricing_feed
    loaded_at_field: _ingested_at
    freshness:
      warn_after:  {count: 26, period: hour}
      error_after: {count: 30, period: hour}

The mute mechanism, which is where the problem is. Kestrel routes alerts to Slack, and the team had built a small wrapper offering a convenience the alerting tool did not:

# alerting/slack_commands.py
def mute(check: str, duration: str = None):
    """/mute <check> [duration]  -- suppress an alert."""
    until = parse_duration(duration) if duration else None
    MUTES[check] = {"until": until, "by": ctx.user, "at": now()}

def is_muted(check):
    m = MUTES.get(check)
    if not m:
        return False
    return m["until"] is None or m["until"] > now()
    #      ^^^^^^^^^^^^^^^^^^^^^ None means FOREVER

parse_duration("48h") returned None — it handled 48h in a later version and, at the time, recognized only 2d and 30m. An unrecognized string produced None, and None meant no expiry.

The failure is None being overloaded. It means "the user did not specify a duration" and "this mute never expires," and the second reading is the dangerous one being reached by an input the user believed specified the first.

⚠️ Failure Mode — a default that is the most dangerous option

duration=None → mute forever is a defaulting decision, and it was made in the direction that maximizes harm on any error.

The general rule: when a parameter is absent or unparseable, default to the SAFEST behaviour, not the most convenient one. For a mute, the safest default is the shortest useful duration — one hour, or until the next scheduled run.

Three properties this wrapper had, each individually reasonable:

An optional duration, because muting "until I fix it" is a real need.

A permissive parser that did not reject unknown formats, because rejecting a command at 04:12 is unhelpful.

None as the sentinel for "indefinite", because it is the obvious Python idiom.

Together they mean a typo grants an indefinite suppression, silently. No single one of the three is the bug.

Two fixes, and you want both:

Reject what you cannot parse. /mute foo 48h with an unparseable duration should reply "I don't understand 48h — try 2d or 4h" rather than doing something. A refusal at 04:12 is annoying; an indefinite mute at 04:12 is 511 days.

Cap the maximum. No mute exceeds seven days without a second person. A control that can be disabled indefinitely by one tired person at 04:12 is not a control.

The Problem

For 511 days the check ran nightly, evaluated correctly, and reported into a suppressed channel.

Two stalls happened during that window:

2025-11-04 → 2025-11-08   pricing feed stalled  4 days
2026-04-22 → 2026-05-01   pricing feed stalled  9 days

During both, dim_product.list_price_cents served the last-known price. Which is not obviously wrong — a price that has not changed is a price that has not changed — and that is exactly why nobody noticed.

What made it wrong is that the April stall coincided with a supplier's quarterly price increase. For nine days, the margin report computed margin using pre-increase costs against post-increase list prices, and the product team made two pricing decisions on it.

Neither decision was reversed. The team's assessment was that both would probably have been made anyway, and the honest version of that assessment is in §"What Happened."

The Analysis

The finding was automatic. mute_audit.py ran for the first time as part of the §23.11 metrics work and printed:

muted checks: 4  (3 older than 30 days)
  pricing_feed_freshness             511 days  data-eng     STALE
  order_items_volume_floor           384 days  NO OWNER     STALE
  session_grain                       47 days  analytics    STALE
  supplier_inventory_schema            2 days  data-eng

Four mutes. Three stale. One with no owner at all.

The second row is worse than the first. order_items_volume_floor is Chapter 19 Case Study 1's assertion — the one added specifically because every other test passes on an empty table. It had been muted for 384 days, and it was muted during a load test in which the team had deliberately run a truncated dataset. The mute outlived the load test by fifty-four weeks.

Step 1: how long, and how do we know? The mute registry stored at (when the mute was set) but not the check's subsequent results, because a suppressed check's output was discarded rather than stored.

That is its own finding, and it is what turned a one-hour investigation into a two-day one: the team could not reconstruct what the check would have said. They had to infer the stalls from _ingested_at gaps in the bronze table.

Step 2: reconstruct the stalls.

SELECT dt, MAX(_ingested_at) AS last_load,
       LEAD(dt) OVER (ORDER BY dt) - dt AS gap_days
  FROM bronze.pricing_feed GROUP BY dt HAVING gap_days > 1;

Which found the two windows above, and confirmed the check would have fired on day two of each.

Step 3: what did the stale prices affect? Lineage from bronze.pricing_feed: stg_pricing → dim_product.list_price_cents → gold.fct_margin → two dashboards.

The margin report was wrong for thirteen days across two incidents, by an amount that varies per product and was not reconstructible with confidence, because the correct list prices for those dates had to be recovered from the supplier's own records.

🔎 Read the Plan — a suppressed check should still record its result

This is the cheapest fix in the case study and it is the one nobody thinks of.

Muting should suppress the notification, not the evaluation. A muted check that still runs and still writes its result gives you:

A history, so "how long has this been failing?" is a query rather than an archaeology project.

An unmute that is informed. Un-muting a check whose last 500 results were failures is a different decision from un-muting one that has been passing.

A dashboard of muted-and-failing, which is the single most useful list in a data platform and almost nobody has it.

text evaluate store result notify normal ✓ ✓ ✓ MUTED (correct) ✓ ✓ ✗ muted (as built) ✓ ✗ ✗ ← 511 days of nothing disabled ✗ ✗ ✗

The difference between rows two and three is one line of code and two days of investigation.

And the fourth row is worth distinguishing deliberately: "disabled" and "muted" should be different states with different approvals. Disabling a check is a decision about whether it is worth running; muting is a decision about whether to be told. Conflating them lets an operational convenience quietly become an architectural one.

The Decision

Five changes.

One: mutes expire, always. No indefinite suppression exists. The maximum is seven days; beyond that requires a second approver and an expiry date.

MAX_MUTE = timedelta(days=7)

def mute(check, duration=None, approver=None):
    d = parse_duration(duration)
    if d is None:
        raise UserError("I don't understand %r. Try 4h, 2d, or 7d." % duration)
    if d > MAX_MUTE and approver is None:
        raise UserError("Mutes over 7 days need a second approver.")
    MUTES[check] = {"until": now() + d, "by": ctx.user, "approver": approver,
                    "reason": require_reason()}

require_reason() is not bureaucracy. A mute with a reason is a mute someone can evaluate later, and the 384-day volume-floor mute would have said "load test" — which anyone reading it after the load test would have removed.

Two: muted checks still evaluate and still store results. The 🔎 callout above.

Three: mute_audit.py runs weekly and posts to the channel, listing every mute with its age, owner, and reason. Not a dashboard — a message, because a dashboard of mutes is a place mutes go to be forgotten a second time.

Four: a mute needs an owner, and the owner is the person who set it, not the team.

Five: the six-box matrix counts a muted check as absent. This one caused an argument, and the argument is the most useful part of the postmortem.

📐 Design Decision — does a muted check count as coverage?

The engineer who raised it had a fair point: a muted check is still written, still correct, and will resume when the mute expires. Counting it as absent makes the coverage metric jumpy and punishes a team for a legitimate operational action.

The counter-argument won, and it generalizes past this case:

The coverage metric answers "would we find out?" During a mute, the answer is no. A check that is not notifying anyone provides exactly the assurance of a check that does not exist, and a metric that says otherwise is measuring the artifact rather than the property.

The jumpiness is the feature. A mute that drops a mart out of full coverage produces a visible change in a number the team looks at weekly, which is precisely the reminder that was missing for 511 days.

And the general form, which is worth carrying beyond mutes: measure whether a control is OPERATING, not whether it EXISTS. The same distinction covers:

a test that exists but cannot fail (Chapter 19 §19.7)

a threshold that cannot fire (Chapter 19 Case Study 2)

an alert routed to an archived channel (Chapter 19 Case Study 2)

a freshness check that is never invoked (Chapter 19 §19.8)

a quarantine that nobody drains (Case Study 1)

Five failures, five chapters, one property. Every one of them passes an audit that counts artifacts, and fails an audit that asks what would happen if the thing went wrong.

What Happened

	Before	After
Mutes	4, three stale, one 511 days	max age 7 days
Indefinite mutes possible	yes	no
Muted checks evaluated	yes	yes
Muted checks' results stored	no	yes
Mute audit	none	weekly, to the channel
Muted check counts as coverage	yes	no

The margin restatement. Thirteen days across two incidents were recomputed with list prices recovered from the supplier. The corrected margin differed by between 0.4 and 3.1 percentage points by product, concentrated in the nine-day April window that overlapped a price increase.

The two pricing decisions were reviewed and left in place. The team's stated reasoning: both were in the direction the corrected numbers also support, and the magnitude of the error was smaller than the range of judgment involved. The postmortem records that this was a judgment and not a calculation, which is the right way to record it — the alternative is a restatement narrative that implies more precision than exists.

Two further findings from the same week:

order_items_volume_floor, muted for 384 days, had nothing to catch. The load-test dataset was never loaded into production. That is luck, and the postmortem says so rather than treating it as evidence the mute was harmless.

session_grain, muted 47 days by the analytics team, was hiding a real failure. It had been firing continuously since it was muted, on a genuine grain violation in silver.sessions introduced by a change to the sessionization overlap (Chapter 18 Case Study 2). Nobody had seen 47 days of failures, because the results were not stored.

That one is the argument for the whole case study: a check that was correct, that fired correctly, on a real defect, every day for 47 days, and told nobody.

Lessons

A control that can be disabled indefinitely by one tired person at 04:12 is not a control. Cap the maximum and require a second approver beyond it.
None overloaded to mean both "unspecified" and "forever" is a defaulting bug waiting for a typo. When a parameter is absent or unparseable, default to the safest behaviour, not the most convenient.
Reject what you cannot parse. A refusal at 04:12 is annoying; an indefinite mute at 04:12 is 511 days.
Muting should suppress the notification, not the evaluation. Store the results. One line of code, and it is the difference between a query and an archaeology project.
Distinguish "muted" from "disabled." Different states, different approvals — one is about being told, the other about whether it is worth running.
A mute needs a reason and an owner. The 384-day mute would have said "load test," and anyone reading that after the load test would have removed it.
Audit mutes weekly, as a message rather than a dashboard. A dashboard of mutes is where mutes go to be forgotten a second time.
A muted check does not count as coverage, because the coverage metric answers "would we find out?" and during a mute the answer is no. The jumpiness is the feature.
Measure whether a control is OPERATING, not whether it EXISTS. A test that cannot fail, a threshold that cannot fire, an alert to a dead channel, a check never invoked, a quarantine never drained — five failures across five chapters, one property, and all five pass an audit that counts artifacts.
A stale price is not obviously wrong, which is why serving one for nine days produced no complaint. It became wrong when it met a price increase.
Record a judgment as a judgment. The two pricing decisions were left in place for reasons the postmortem states plainly, rather than dressed as a calculation.
One of the four mutes was hiding 47 days of real, daily, correct failures on a genuine defect — and nobody could see them, because a muted check's results were discarded.

Questions for Discussion

The on-call engineer's action was correct and the outcome was 511 days. Where, precisely, should the responsibility sit, and what does your answer imply about how to write the postmortem?
parse_duration returning None for an unknown format is a common Python idiom. Where else does that idiom sit next to a dangerous default in code you own?
The team argued about whether a muted check counts as coverage. Argue the losing side as strongly as you can. What does the coverage metric lose by being jumpy?
Storing a muted check's results costs almost nothing and would have saved two days. Why do you think almost no alerting system does it by default?
order_items_volume_floor was muted for 384 days and had nothing to catch. How should a postmortem treat a near-miss that produced no harm — and what is the risk of treating it as harmless?
session_grain was firing daily on a real defect for 47 days. What would have surfaced it, other than the mute audit?
§"Design Decision" lists five failures across five chapters that all pass an artifact-counting audit. Design a single audit that catches all five. What does it cost, and what would still get past it?