Case Study 1: Two to Add, One to Change, One to Destroy

"The pull request renamed a database. Everyone reviewed the rename. Terraform read it as: delete this database, then make an empty one with the new name."

Executive Summary

A pull request standardized Kestrel's Snowflake naming: ANALYTICS became KESTREL_ANALYTICS, alongside three other cosmetic changes. Two engineers approved it. The diff was four lines and every one of them was a string.

snowflake_database.name is a ForceNew attribute. Terraform's plan said:

Plan: 2 to add, 1 to change, 1 to destroy.

The one to destroy was the production analytics database, containing every gold model. The apply ran at 14:12 on a Wednesday and completed in nine seconds.

Recovery took six hours and was possible only because Snowflake's Time Travel retained the dropped database for one day, and because someone thought to check within that day. The margin was under seven hours.

Skills applied: reading a plan's resource lines rather than its summary (§28.3); prevent_destroy; the moved block (§28.12's 🧭 note); and the general shape of a destructive change that looks cosmetic.

Background

The change, in full:

 resource "snowflake_database" "analytics" {
-  name    = "ANALYTICS"
+  name    = "KESTREL_ANALYTICS"
   comment = "gold + silver models"
 }

 resource "snowflake_warehouse" "transform" {
-  name = "TRANSFORM_WH"
+  name = "KESTREL_TRANSFORM_WH"
   ...
 }

A naming-convention cleanup, of exactly the kind that gets merged on a quiet afternoon. It was reviewed by two people, both of whom read it as a rename, because that is what it is.

What Terraform does with a rename depends entirely on whether the provider marks the attribute ForceNew, and the review has no way to see that — it is a property of the provider, documented on a page nobody has open, and it differs between resource types in the same provider:

snowflake_database.name        ForceNew   → destroy and recreate
snowflake_warehouse.name       ForceNew   → destroy and recreate (but empty)
snowflake_database.comment     in-place

Both renames forced a replacement. Only one of them mattered, because a warehouse contains no data and a database contains everything.

The Problem

The plan output, in full, as it appeared in the CI log:

Terraform will perform the following actions:

  # snowflake_database.analytics must be replaced
-/+ resource "snowflake_database" "analytics" {
      ~ name = "ANALYTICS" -> "KESTREL_ANALYTICS" # forces replacement
    }

  # snowflake_warehouse.transform must be replaced
-/+ resource "snowflake_warehouse" "transform" {
      ~ name = "TRANSFORM_WH" -> "KESTREL_TRANSFORM_WH" # forces replacement
    }

Plan: 2 to add, 1 to change, 1 to destroy.

Everything needed is in that output. must be replaced, -/+, and # forces replacement — three separate signals, all present, all correct.

And the summary line, which is what people read, says 1 to destroy — which is true, and does not say what, and is easy to read as the routine churn every plan contains.

⚠️ Failure Mode — the summary line is the only part that gets read

Plan: 2 to add, 1 to change, 1 to destroy appears at the bottom of every plan, and most plans contain a destroy that is completely routine — a security-group rule being replaced, a null resource, a rotated attachment.

So the summary line trains you to expect a destroy, and by the time a plan contains one that matters, the count carries no information at all.

Three properties made this specific instance nearly invisible:

• The count was small. One. A plan proposing to destroy fourteen things gets read.
• The diff was cosmetic. Four string literals, in a PR titled "standardize naming."
• The plan was in a CI log, collapsed, and the summary was in the PR check's title.

The fix is not "read the plan more carefully." That is Chapter 27 §27.10's rejected instruction — if a defect is invisible in the artifact under review, diligence is not the fix.

The fix is that the artifact under review must show it. Two independent controls, and Kestrel now has both:

• prevent_destroy on the resource, which makes the apply fail rather than proceed. It is one block, and it converts a catastrophe into an error message.
• A CI check on the plan JSON that surfaces the resource-level detail as a blocking comment, above the fold. code/plan_review.py.

Neither is sufficient alone. prevent_destroy can be removed in the same pull request, and a CI check can be overridden — which is why two, and why they fail in different ways.

The Analysis

The apply ran from CI on merge, which was the team's normal flow and had been for a year.

14:12:03  Terraform apply
14:12:04  snowflake_database.analytics: Destroying... [id=ANALYTICS]
14:12:07  snowflake_database.analytics: Destruction complete after 3s
14:12:07  snowflake_database.analytics: Creating...
14:12:09  snowflake_database.analytics: Creation complete after 2s
14:12:12  Apply complete! Resources: 2 added, 1 changed, 1 destroyed.

Nine seconds, and Apply complete! — a success message, because from Terraform's point of view it did exactly what it was asked.

What was lost: every gold and silver model. 90 tables. fct_order_item, dim_customer, gold.daily_revenue, and everything downstream of them.

What was not lost: bronze, which is in a separate database, and the lake, which is in S3. So the data was recoverable by rebuilding — which would have taken about four hours — and the question was whether anything was needed that could not be rebuilt.

Step 1: what noticed? Not the apply. The first signal was a dashboard error at 14:31, nineteen minutes later, when someone opened the executive report.

And this is worth pausing on: Chapter 25's monitoring is built around a nightly cadence — freshness against a 06:00 deadline, duration against a trailing median, a margin. None of it fires at 14:12 on a Wednesday, because none of it was designed for a mid-afternoon catastrophe.

Step 2: what is recoverable, and for how long? This determined everything, and it is Chapter 26 §26.5's retention check being run under pressure rather than in advance:

-- Snowflake retains a dropped database for its Time Travel period.
SHOW DATABASES HISTORY LIKE 'ANALYTICS';
-- dropped_on: 2026-09-16 14:12:07  retention_time: 1 day

One day. DATA_RETENTION_TIME_IN_DAYS was at the account default of 1, on an Enterprise account where it could have been up to 90.

$$\text{recovery window} = 24\ \text{hours from } 14{:}12 \Rightarrow \text{deadline } 14{:}12 \text{ tomorrow}$$

UNDROP DATABASE ANALYTICS;     -- 14:47. Complete in 4 seconds.

Four seconds to restore what took nine seconds to destroy, and the entire incident's risk was contained in the gap between 14:12 and whenever somebody thought to run that.

Step 3: what did the six hours go on? Not the restore. The reconciliation:

• Confirming all 90 tables were present and at the expected row counts.
• Re-running the day's incremental loads, which had been skipped.
• Re-granting permissions — the UNDROP restored the database and not the grants, which were managed by a separate Terraform resource and had to be re-applied.
• Verifying that the 14:12–14:47 window had not written anything into the new empty database that now needed reconciling.

🔎 Read the Plan — UNDROP restored the data and not the access

The detail that turned a forty-minute incident into a six-hour one, and it generalizes past Snowflake.

A restore returns the object. It does not return everything about the object's context — grants, tags, policies, and, in some systems, replication or streams.

```sql -- After UNDROP: the tables are there. SELECT COUNT(*) FROM ANALYTICS.GOLD.FCT_ORDER_ITEM; -- 6,483,117 ✓

-- And nothing can read it. SHOW GRANTS ON DATABASE ANALYTICS; -- 0 rows ```

Terraform had the grants, in a different resource, and re-applying them was straightforward once somebody realized. Finding that out took ninety minutes, most of it spent debugging permission errors from a BI tool that reported them as connection failures.

The runbook item this produced, and it belongs in yours: "after any restore, verify grants, tags, and policies separately from data. The restore command does not." Then test it — Chapter 26 Case Study 2's drill, which Kestrel then ran on this runbook and found two more implicit steps.

The Decision

Five changes, and the first two are the ones that matter.

One: prevent_destroy on every stateful resource.

resource "snowflake_database" "analytics" {
  name    = "ANALYTICS"
  lifecycle { prevent_destroy = true }
}

The apply now fails with an explicit error naming the resource, and removing the block is a separate, visible, reviewable line in a diff.

Two: plan_review.py between plan and apply, blocking on any destroy or replacement of a stateful resource. Above the fold, in the pull request, with the resource named.

Three: DATA_RETENTION_TIME_IN_DAYS raised from 1 to 7, and set in Terraform so it cannot drift back.

📐 Design Decision — why 7, and not 90

The Enterprise account allows up to 90 days of Time Travel, and the obvious response to a near-miss with a 24-hour window is to take all of it.

They chose 7, and the reasoning is the one Chapter 26 §26.5 asks for:

Time Travel is not free. Snowflake charges for the storage the retained versions occupy, and on a database rebuilt nightly that is substantial — a rough estimate put 90 days at $1,900 a month against $150 for 7.

And the question retention answers is "how long until somebody notices." Kestrel's realistic detection time for a destroyed database is minutes — a dashboard breaks. Seven days is more than two orders of magnitude of margin on that.

Which is not the same question Chapter 24 Case Study 1 asked, where a March write was damaged in August and needed 141 days. The two incidents want different retentions because they are different failures, and the honest conclusion Kestrel recorded is that Time Travel is the right control for a destruction and the wrong one for a slow corruption — for which the source system's 18-month retention is the actual recovery path.

Set retention from the failure it is protecting against, and name that failure in the comment.

Four: moved blocks for genuine renames. §28.12's 🧭 note — a rename in configuration without a destroy:

moved {
  from = snowflake_database.analytics
  to   = snowflake_database.kestrel_analytics
}

This handles renaming the Terraform address. Renaming the object is still a replacement, and the team's conclusion — do not rename production databases for cosmetic reasons — is recorded as a decision rather than left implicit.

Five: applies to the stateful state file require a second approver. §28.8's split, which was implemented in the same fortnight for exactly this reason.

What Happened

The margin is the number that stayed with the team. The UNDROP was available for one day, it was used within 35 minutes, and nobody involved knew what the retention was until they looked it up during the incident.

Three further findings:

Nothing in Chapter 25's monitoring fires at 14:12. Every check was built around a nightly cadence and a 06:00 deadline. A freshness check that runs hourly was added — cheap, and it would have signalled at 15:00 rather than depending on a person opening a dashboard.

Twelve other resources had no prevent_destroy, including both lake buckets. A bucket replacement would not have been recoverable at all, because S3 versioning does not survive a bucket deletion.

The naming convention was abandoned. Not because it was wrong, but because the team could not identify a benefit that justified any risk to a production database, and writing that down was the outcome: "cosmetic changes to stateful resource names are not worth doing. If a name is genuinely wrong, that is a migration, not a rename."

Lessons

1. A rename can be a destruction, and whether it is depends on a ForceNew flag in the provider that is not visible in the diff.

2. The summary line is the only part that gets read, and most plans contain a routine destroy — so the count carries no information by the time it matters.

3. "Read the plan more carefully" is not a fix. The artifact under review must show it.

4. Two independent controls, failing differently: prevent_destroy makes the apply fail; a CI check on the plan JSON makes it visible before merge. Either can be bypassed; both being bypassed requires two deliberate acts.

5. Apply complete! is a success message for doing exactly what was asked.

6. Nothing in a nightly-cadence monitoring system fires at 14:12 on a Wednesday. The first signal was a person opening a dashboard nineteen minutes later.

7. A restore returns the object, not its context. Grants, tags, and policies came back separately, and finding that out took ninety minutes of debugging permission errors reported as connection failures.

8. Set retention from the failure it protects against, and name that failure. Seven days for a destruction (detected in minutes); Chapter 24 Case Study 1's slow corruption needed 141 and Time Travel is the wrong control for it.

9. Nobody knew the retention until they looked it up during the incident. It should have been in a runbook, and Chapter 26 §26.5 says so.

10. A bucket replacement is not recoverable at all — versioning does not survive the bucket.

11. The cosmetic change was abandoned and the reason was written down. "If a name is genuinely wrong, that is a migration, not a rename."

Questions for Discussion

1. Two engineers reviewed a four-line diff of string literals. What could either have done, and is it reasonable to expect it?

2. The plan output contained three separate correct signals. Why did none of them work, and what does that suggest about signal design generally?

3. prevent_destroy and the CI check can both be bypassed. Is "two controls that fail differently" genuine defence, or two-thirds of a ritual?

4. Retention was set at 7 days from a detection time of minutes. Argue for 30. What would you be buying?

5. Nothing fired at 14:12 because the monitoring assumed a nightly cadence. What else in your systems assumes a schedule that a failure does not respect?

6. The UNDROP window was 24 hours and was used within 35 minutes. What would have happened at 02:00 on a Saturday, given Chapter 26 §26.1's paging policy?

7. The team abandoned a naming convention rather than accept any risk to a production database. Is that proportionate, or is it an over-correction from one incident?