Case Study 2: The Fix That Kept Getting Undone

"Four times. The same incident, the same fix, and every time somebody's unrelated pull request put it back."

Executive Summary

Kestrel's TRANSFORM_WH warehouse was too small for the nightly build after two years of growth. On four separate occasions an on-call engineer resized it from MEDIUM to LARGE at around 05:20 to meet the 6am SLA — which was the correct action, and Chapter 26 §26.8 pre-authorizes it.

Each time, the next unrelated terraform apply silently put it back. The apply's plan contained the reversion; the plan was six resources long; nobody read the line about a resource their pull request had not touched.

Four incidents, four correct mitigations, four silent reversions, over eleven weeks — and the fourth was the one that finally got investigated, not because anyone noticed the pattern but because the on-call engineer that morning had also been the one on the first occasion and recognized it.

The fix was to stop treating drift as misbehaviour: a nightly drift check that posts to a channel, and a plan reviewer that flags a reversion specifically.

Skills applied: drift as information (§28.10); reading a plan's resource lines (§28.3); pre-authorized on-call actions (Chapter 26 §26.8); and the general problem of two correct actions composing into a wrong outcome.

Background

The underlying problem was real and known. The nightly build's duration had been creeping (Chapter 25 Case Study 1), and on heavy nights a MEDIUM warehouse could not finish before 06:00.

The proper fix — resize permanently, or reduce the build's work — was on the backlog, correctly prioritized behind other things, and would have taken a decision about cost that nobody had made.

The mitigation was one click:

ALTER WAREHOUSE TRANSFORM_WH SET WAREHOUSE_SIZE = 'LARGE';

Chapter 26 §26.8's pre-authorization table explicitly permits this. The on-call engineer may change compute to meet an SLA; they may not change business logic. This is the system working.

And Terraform holds warehouse_size = "MEDIUM", because that is what it was set to eighteen months earlier.

The Problem

week 1   05:22  resize to LARGE. SLA met at 05:56. Incident closed.
         11:40  unrelated PR (a new S3 bucket). apply. → back to MEDIUM.

week 4   05:18  resize to LARGE. SLA met at 05:51.
         16:05  unrelated PR (an IAM policy). apply. → back to MEDIUM.

week 8   05:31  resize to LARGE. SLA MISSED at 06:04.
         09:20  unrelated PR (a tag). apply. → back to MEDIUM.

week 11  05:19  resize to LARGE. SLA met at 05:54.
                ← the engineer recognizes it. Investigation starts.

Every one of the eight actions in that table is correct in isolation. The resizes were pre-authorized mitigations; the applies were reviewed pull requests doing what their authors intended.

The plan, every time, contained the reversion:

  # snowflake_warehouse.transform will be updated in-place
  ~ resource "snowflake_warehouse" "transform" {
      ~ warehouse_size = "LARGE" -> "MEDIUM"
    }

  # aws_s3_bucket.exports will be created
  + resource "aws_s3_bucket" "exports" { ... }

Plan: 1 to add, 1 to change, 0 to destroy.

One line, in a plan the author skimmed for the thing they had asked for.

⚠️ Failure Mode — two correct actions composing into a wrong outcome

This is the shape worth naming, because no individual review or process step catches it.

The on-call engineer did the right thing. They had a pre-authorization, an SLA in forty minutes, and a one-click mitigation. Recording it in Terraform at 05:22 was not available — that is a pull request, a review, and an apply, during an incident.

The pull-request author did the right thing. They wrote a bucket, reviewed the plan for their bucket, and applied it.

Nobody was wrong, and the outcome was wrong four times.

The general form: a shared, converging system where one actor changes reality and another actor restores the declared state. It appears everywhere:

  • Terraform reverting a console change ← this
  • A config-management tool reverting a hotfix on a host.
  • A dbt run --full-refresh reverting a manual data patch.
  • An auto-scaler reverting a manual scale-up.
  • A deployment reverting a debug flag someone set in production.

In every case the second actor is doing its job, which is why "be more careful" does not apply to anyone, and why the fix has to be making the divergence visible to a person who can decide.

Convergence without notification is the defect.

The Analysis

Step 1: how long had this been happening? The Snowflake query history answered it in one query, and the answer was worse than four:

SELECT query_text, user_name, start_time
  FROM snowflake.account_usage.query_history
 WHERE query_text ILIKE '%ALTER WAREHOUSE%SET%WAREHOUSE_SIZE%'
   AND start_time > dateadd(month, -6, current_timestamp)
 ORDER BY start_time;

resizes to LARGE   (people, ~05:20)                  7
resizes to MEDIUM  (the terraform service account)   7

Seven, not four. Three had happened in the first three months and had not been recognized as a pattern because different engineers were on call.

Step 2: what did it cost? Two things, and the second is the larger.

One missed SLA, in week 8, when the resize happened at 05:31 and the build did not finish until 06:04.

And 4.5 person-hours of incident time, seven times over — because each occurrence was investigated from scratch. Nobody had a record that this had happened before, since each incident's postmortem concluded "warehouse too small; resized; the permanent fix is on the backlog."

$$7 \times 4.5\ \text{hours} = 31.5\ \text{hours of repeated diagnosis}$$

Which is the cost of the pattern being invisible, not of the underlying capacity problem.

🏭 From the Pipeline — an incident that recurs is a different incident

Seven postmortems, each individually reasonable, each concluding the same thing, and none of them referencing the previous one.

The mechanism is that a postmortem is written about an occurrence, and the question "has this happened before?" is not on any template — including Chapter 26 §26.9's, which this incident caused to gain a field:

```markdown

Recurrence

  • [ ] Search the incident log for this symptom. Prior occurrences: ______
  • [ ] If this is the 2nd occurrence, the action item is NOT the same mitigation. Say what is different about doing it again. ```

"If this is the second occurrence, the action item is not the same mitigation." That sentence is the whole fix, and it is deliberately blunt: a mitigation applied twice is a process, and a process should be either automated or eliminated.

The search is the hard part in practice, because incident records are prose and symptoms are described differently by different people. Kestrel's partial answer is a required symptom tag from a controlled list — sla-missed, source-late, capacity, data-wrong — which is crude, is searchable, and would have surfaced the pattern at occurrence two rather than seven.

Step 3: why did nobody read the plan line? Because the plan was, in the author's mind, about their bucket — and a resource they had not touched appearing in the plan is not something the format distinguishes. A plan is a flat list; there is no "changes you asked for" and "changes that come along."

The Decision

Four changes.

One: a nightly drift check that posts to a channel.

# 07:00. Exit 2 means drift.
terraform plan -detailed-exitcode -refresh-only -no-color > drift.txt
case $? in
  0) ;;                                        # no drift
  2) post_to_channel "drift detected" drift.txt ;;
  *) post_to_channel "drift check FAILED" ;;   # Ch. 27 CS2: not a pass
esac

07:00 is deliberate: after the 05:00 window, so the on-call engineer sees their own change named in the channel while it is still their morning.

Two: plan_review.py flags a reversion specifically. A change to a size, a count, or a capacity attribute, on a resource not otherwise modified by this pull request, gets its own line:

[WARN ] snowflake_warehouse.transform   warehouse_size changes 'LARGE' -> 'MEDIUM'.
        If nobody in this change asked for that, it is a REVERSION of a console
        change -- possibly an on-call mitigation. sec 28.10.

Three: the incident template gains a "did you change anything outside code?" field, and a recurrence field.

Four: the actual capacity decision was made. The warehouse went to LARGE permanently, in Terraform, with the cost stated:

$$\text{MEDIUM} \to \text{LARGE} = 4 \to 8\ \text{credits/hour}$$

At the frozen $2.00/credit and about 2.6 hours of nightly build:

$$(8 - 4) \times 2.6 \times \$2.00 \times 365 = \$7{,}592\ \text{a year}$$

📐 Design Decision — the cost that nobody would decide

The most interesting part of this incident is that the permanent fix cost $7,592 a year and took eleven weeks to approve, while the mitigation cost 31.5 hours of engineering time and was applied seven times without anyone approving anything.

The asymmetry is structural, not a failure of judgment:

  • $7,592 is a budget line. It requires a decision, an owner, and a conversation with whoever owns the budget.
  • 31.5 hours of on-call time is nobody's line item. It is absorbed, it is invisible, and it never comes up in a planning meeting.

So the expensive option was chosen repeatedly by default, and the cheap one required a meeting.

What changed it was pricing the mitigation. Once "we have spent 31.5 hours and missed one SLA" sat next to "$7,592," the decision took a day — which is Chapter 25 Case Study 1's lesson arriving in a different form: the argument that works supplies a number and a consequence.

And the honest counterweight, which the team recorded: LARGE is not obviously right either. The permanent fix that would have been better is Chapter 25 Case Study 1's margin work — deleting unread models, fixing the sensors, the check_cols explosion — which recovered 168 minutes and cost nothing per year. They did both, and the warehouse was later returned to MEDIUM once the margin work landed.

A capacity increase is the fastest fix and the one you should be least satisfied with.

What Happened

Before After
Reversions 7 over 6 months 0
Repeated diagnosis 31.5 hours —
SLAs missed to this cause 1 0
Drift detected never nightly, to a channel
Warehouse size MEDIUM (declared) / LARGE (reality) one answer

The drift check found four other things in its first week, none of them related and all of them worth knowing:

  • An S3 lifecycle rule added in the console eight months earlier, transitioning a prefix to Glacier after 60 days. The prefix was read by a quarterly job, which had been paying restore fees.
  • An IAM role with an extra policy attached during an incident in the spring and never removed.
  • A Snowflake resource monitor that had been suspended manually and never resumed, so a spend cap had not been enforced for five months.
  • A tag removed by someone tidying the console, which had quietly excluded a warehouse from Chapter 25 §25.6's cost attribution.

None was a Terraform problem. All four were console changes that were correct at the time, made by people acting reasonably, and invisible for months because nothing compared reality to the declaration.

And the framing was the change that made it work. The drift message says "drift detected" rather than "unauthorized change", and the team's note is explicit: "if this reads as an accusation, people will make undocumented changes instead of documented ones, and we will have made it worse."

Lessons

  1. Two correct actions can compose into a wrong outcome, and no individual review catches it. The general form: one actor changes reality, another restores the declared state.

  2. Convergence without notification is the defect — not the console change, and not the apply.

  3. The reverting apply's plan contained the reversion every time, on a resource the author had not touched. A plan is a flat list; it does not distinguish "what you asked for" from "what comes along."

  4. Drift found nightly is information, not a violation. A team that treats it as misbehaviour gets undocumented changes instead of documented ones — so the message wording matters.

  5. Post the drift check after the on-call window, so the engineer sees their own change named while it is still their morning.

  6. An incident that recurs is a different incident. Seven postmortems, each reasonable, none referencing the previous. "Has this happened before?" is not on any template, and a controlled symptom tag makes it searchable.

  7. "If this is the second occurrence, the action item is not the same mitigation." A mitigation applied twice is a process; automate it or eliminate it.

  8. A budget line requires a decision; engineering hours do not. $7,592 a year took eleven weeks; 31.5 hours was spent without anyone approving anything. Pricing the mitigation is what moved it.

  9. A capacity increase is the fastest fix and the one to be least satisfied with. The margin work recovered 168 minutes for nothing per year, and the warehouse went back to MEDIUM afterwards.

  10. The drift check found four unrelated things in its first week, including a lifecycle rule costing restore fees and a suspended spend cap. All were correct decisions at the time, invisible for months.

Questions for Discussion

  1. The on-call engineer could not reasonably have recorded the change in Terraform at 05:22. What is the lightest possible way to record it that they could have done?

  2. §"The Analysis" notes that a plan does not distinguish requested changes from incidental ones. Design that distinction. What would it require?

  3. The drift message says "drift detected" rather than "unauthorized change." Is that a meaningful difference or a euphemism? What would tell you?

  4. Seven postmortems missed the pattern because each was about an occurrence. What is the cheapest change that surfaces recurrence, and what does it cost in friction?

  5. The mitigation cost 31.5 invisible hours and the fix cost $7,592 visible dollars. Where else does your organization make that trade by default?

  6. The team eventually did the margin work and returned the warehouse to MEDIUM. Was the LARGE period a waste, or the thing that bought time for the better fix?

  7. Four unrelated drift findings surfaced in the first week. What do you expect a drift check would find in your systems this week? Write the number down before running it.