Case Study 1: The Quarter That Got Worse After It Ended

"We reported 100% correctness for Q2. In September we found out Q2 had been 88.9%, and the argument was about whether we were allowed to say so."

Executive Summary

Kestrel introduced three SLOs in April 2026 — freshness, completeness, and correctness, the last scored retrospectively: a day counts correct until a defect is found that proves otherwise.

Q2 was reported at 100% correctness. In September, Chapter 20's watermark defect was found, having run from 2026-08-03 to 2026-08-12. Ten days of Q3 were revised from correct to not-correct, and the same revision applied to Q2 for a different defect took its correctness from 100% to 88.9%.

The technical work was ten minutes. The argument that followed took three weeks and is the case study: an SLO that can be revised downward after a quarter closes is an SLO that makes a team look worse the more carefully it investigates.

The resolution — separating the number from the report, and publishing both — is the useful part, and it generalizes past SLOs to every retrospective measure a team keeps.

Skills applied: SLI/SLO design (§26.2); retrospective correctness; error budgets (§26.3); and the organizational half of measurement, which is where most measurement systems fail.

Background

The three SLOs, adopted in April:

Why correctness got the loosest objective was a deliberate decision and, in retrospect, the only part of the original design that survived unchanged: it is the SLI most likely to be revised downward, so it needs the most budget. A tight correctness SLO would be blown by the first discovery, which is §26.2's ⚠️ callout applied before the fact.

The definition, as written:

A day is scored not correct if a Sev-1 or Sev-2 data defect is later found to have affected data published on that day, regardless of when the defect was discovered.

Everyone agreed with that sentence in April, and nobody had thought through the consequence.

The Problem

Q2 closed on 30 June and was reported on 8 July:

Q2 2026 data platform SLOs          (90 days; each miss is 1.11 points)
  freshness      100.0%  ✅ (objective  99.5%)
  completeness   100.0%  ✅ (objective  99.0%)
  correctness    100.0%  ✅ (objective  97.0%)

All three met, with no misses at all. The report went to the leadership meeting.

Note the granularity, because it matters later: over 90 days a single bad day is 1.11 percentage points, so a 99.5% freshness objective is met at 90/90 and missed at 89/90. There is no such thing as 99.6% in a 90-day window, and an objective set between two achievable values is worth noticing before you adopt it.

On 12 September, Chapter 20's Black Friday reconciliation found the watermark defect — orders lost because updated_at had no lookback — and the investigation established that it had been running since the previous December.

Which meant Q2 was not 100% correct. It had been 100% undiscovered.

Applying the definition:

$ python slo.py q2_days.json --incidents incidents.json --sli correctness

applied 1 incident(s): 10 day(s) revised to not-correct
SLI               days    bad    attain    budget      left  burn/7d
----------------------------------------------------------------------
correctness         90     10    88.89%      2.70     -7.30     0.00  BLOWN

Q2's correctness was 88.9% against a 97% objective, and the report saying 100% had been in a leadership deck for two months.

⚠️ Failure Mode — a retrospective measure punishes investigation

Here is the shape of the problem, and it is worse than it first looks.

The watermark defect was found because somebody investigated a 2.1% Black Friday variance. Nobody had to. The reconciliation had been passing at 0.02–0.06% for eight months, and the person who pulled the thread turned a quarter that read as perfect into one that missed its objective by eight points.

So the measure, as designed, creates a direct incentive not to look.

And the incentive is subtle rather than corrupt. Nobody would decline to investigate a defect to protect a number. What happens instead is quieter: the investigation is deprioritized, the variance is "probably timing," and the thread does not get pulled this quarter. Chapter 23's reconciliation tolerance is exactly this mechanism at the level of a threshold, and this is the same mechanism at the level of a team's reported performance.

Three responses were proposed, and two are wrong in instructive ways:

"Score correctness at discovery, not occurrence." A defect found in September counts against September. This removes the disincentive and destroys the measure — it now says how often you find things, which is maximized by finding nothing.

"Freeze a quarter once reported." Clean, auditable, and it means the reported number is known to be wrong and known to stay wrong. The measure becomes a report of what was believed, which is not a measure of anything.

"Publish both." §"The Decision". It is more work and it is the only one that keeps the number honest.

The Analysis

Step 1: how much of the reported history is affected? Applying the definition to every known defect, across five quarters:

quarter   as reported   as revised   defects found later
2025-Q3      100.0%        94.4%     1 (the duplicate-rows incident)
2025-Q4      100.0%        98.9%     1
2026-Q1       98.9%        87.8%     2 (incl. the watermark)
2026-Q2      100.0%        88.9%     1 (the watermark)
2026-Q3       97.8%        87.8%     2 (watermark, check_cols)

Every quarter got worse. Not one revision moved a number up, which is a property of the definition rather than of the platform: defects are only ever discovered, never un-discovered.

Step 2: is the platform getting worse? The revised series says 94.4 → 98.9 → 87.8 → 88.9 → 87.8, which looks like decline.

It is not. The later quarters have had less time to be investigated:

                  months of hindsight    defects found so far
2025-Q3                        14                          1
2026-Q3                         0                          2

Q3 2026 already has two, with zero months of hindsight. By the standard the older quarters were scored against, it will have more.

🔎 Read the Plan — a retrospective measure is not comparable across time until it stabilizes

This is the analytic trap, and it is the same one every "incidents per quarter" chart contains without anyone noticing.

The recent past always looks better than the distant past, because the distant past has been investigated for longer. A chart of revised correctness by quarter therefore shows a spurious improving trend if you read it left to right and a spurious declining one if the discovery rate happens to be rising.

The fix is to report the measure at a fixed age, not at the present:

text correctness, as scored SIX MONTHS after each quarter closed 2025-Q3 94.4% 2025-Q4 98.9% 2026-Q1 87.8% 2026-Q2 88.9% ← reportable from 2026-12 2026-Q3 — ← not yet comparable

Six months is a judgment, chosen because Kestrel's median time-to-discovery for a Sev-1 data defect was 47 days and the 90th percentile was 5 months. The number is defensible because it came from the discovery distribution, which is a thing you can measure once you are recording discoveries at all.

The general rule: any measure that can be revised downward by later information must be reported at a fixed lag, or it is not a time series. It applies to incident counts, to defect rates, to anything scored by a subsequent discovery — and almost nobody does it.

Step 3: what does the error budget say? Once revised, correctness was blown in four of five quarters. The budget was not too small — the platform was genuinely worse than 97%, and the original objective had been set from an attainment figure that was itself un-revised.

Which is a circular error worth naming: the SLO was set from observed attainment, and observed attainment was inflated by undiscovered defects. Setting an objective from a measure that is biased upward produces an objective that is biased upward.

The Decision

Four changes, and the first is the one that resolved the argument.

One: separate the number from the report.

Nobody's past report is retroactively made wrong, and the live number is honest. The two artifacts answer different questions and the conflict was entirely from making one artifact do both.

Two: publish the revisions.

CORRECTNESS REVISIONS
2026-09-14  2026-Q2  100.0% → 88.9%  INC-91 watermark, 10 days
            2026-Q1   98.9% → 87.8%  INC-91 watermark, 10 days
            reported by: the engineer who investigated the BF variance

Naming the person who caused the revision was deliberate, and it was the change that most shifted the team's behaviour.

📐 Design Decision — crediting the revision, not just recording it

The obvious way to log a revision is by defect. Kestrel logs it by investigator, and the reasoning is a direct answer to the ⚠️ callout's incentive problem.

If finding a defect makes the numbers worse and nothing else, finding defects is thankless. So the revision log names who found it, the quarterly review reads the list out, and "caused three revisions this quarter" is a positive line in a performance conversation.

It is a small thing and it inverts the incentive completely. The number still goes down; the person who made it go down is now visibly the person doing the most valuable work on the platform.

The obvious objection was raised and is worth recording: does this encourage inflating minor issues into Sev-2s? In two years it has not, and the team's view of why is that the severity definition is checkable by anyone and the reviews are public. An incentive to over-report is constrained by the same visibility that makes the credit work.

The general principle: when a measure creates a disincentive to investigate, add an explicit counter-incentive rather than weakening the measure. Weakening it is the reflex, and it is how most retrospective measures quietly become reports of what was believed.

Three: re-set the objectives from the revised series, not the reported one. Correctness moved from 97% to 92%, which is lower, honest, and — per §26.2's test — a number that has been close in both of the last two quarters.

Four: report the discovery distribution alongside the SLO, because it is what makes the fixed lag defensible and because it is a platform metric in its own right:

time to discovery, Sev-1/Sev-2 data defects, trailing 8 quarters
  median   47 days
  p90       5 months
  max      14 months     ← 2025-Q3's duplicate-rows incident

That last row is Chapter 1's incident, and having it in the same table as the SLO makes the argument for everything in Chapters 23 and 25 without anyone having to make it.

What Happened

The median time to discovery fell to 31 days, which the team attributes mostly to Chapters 23 and 25's work rather than to the incentive change — and says so, because the two were done in the same period and cannot be separated.

Two things that did not go well, recorded because the postmortem template requires it:

The three-week argument was expensive and it happened at the worst possible time, during an incident's aftermath, with the people involved already tired. The definition should have been stress-tested in April by asking one question: what happens when we find something?

One person stopped reporting minor Sev-2s for about a month, before the credit change landed. They raised it themselves in a retrospective, which is the only reason anyone knows. The incentive was real and it was operating before anyone had named it.

Lessons

1. A retrospective correctness measure is the only SLI that measures the failures in this book, and it is the only one that can make a closed quarter worse.

2. It creates a direct incentive not to investigate, and the incentive is quiet: not a refusal, a deprioritization.

3. Scoring at discovery destroys the measure — it becomes "how often do you find things," which is maximized by finding nothing. Freezing a quarter destroys it differently — it becomes a record of what was believed.

4. Publish both. The report is frozen and marked as known at a date; the number is live with a change log. The conflict came entirely from making one artifact do two jobs.

5. Log revisions by investigator, and read the list out. "Caused three revisions this quarter" should be a positive line. A small change that inverts the incentive completely.

6. When a measure creates a disincentive to investigate, add a counter-incentive rather than weakening the measure. Weakening it is the reflex.

7. A revisable measure is not a time series until it stabilizes. Report at a fixed lag derived from your own discovery distribution — Kestrel's median 47 days, p90 five months, so six.

8. The recent past always looks better than the distant past under such a measure. Reading the chart left to right produces a spurious trend in whichever direction the discovery rate happens to move.

9. An objective set from an inflated attainment figure is inflated. Correctness went from 97% to a lower, honest 92% once set from the revised series.

10. Publish the time-to-discovery distribution next to the SLO. It makes the fixed lag defensible and it argues for everything in Chapters 23 and 25 without anyone having to.

11. Stress-test a definition when you adopt it, with one question: what happens when we find something? Three weeks of argument during an incident's aftermath was the cost of not asking it in April.

Questions for Discussion

1. The definition was agreed by everyone in April and nobody thought through the consequence. What would "stress-testing a definition" look like as a routine practice?

2. Crediting revisions by investigator inverts the incentive. What else could it distort, and how would you detect that?

3. The team publishes both a frozen report and a live number. Is that transparency or an opportunity to cite whichever is convenient? What would prevent the second?

4. Six months was chosen from the discovery distribution. Defend a different lag. What does three months lose, and what does twelve cost?

5. One person stopped reporting minor Sev-2s for a month and raised it themselves. What made that possible, and would it have surfaced in your team?

6. The revised series shows every quarter getting worse. How would you present that to leadership without it reading as a platform in decline?

7. Chapter 20's watermark defect was found because someone investigated a variance nobody required them to. How many such threads are currently unpulled in your systems, and what would make pulling one rewarding?