Chapter 28: Infrastructure as Code

"The plan said it would replace one resource. Replacing that resource meant dropping the database. Nobody read past the summary line."

Overview

Everything in this book runs on something: a warehouse, a bucket, a cluster, a scheduler, a database. This chapter is about defining that something in files rather than in a console, and about the specific ways that goes wrong for a data platform.

The general argument for infrastructure as code is well made elsewhere and this chapter does not repeat it. What it does instead is take the three questions a data team actually gets stuck on:

Where is the boundary? Terraform can create a warehouse, a role, a grant, a schema — and a table. Should it? §28.3, and the answer is a line rather than a rule.

How much of this do we need? A four-person team running Kubernetes is running a second platform. §28.5 is the honest version.

What happens when someone changes something at 05:00? They will, correctly, and forbidding it is not available. §28.10.

Chapter 19 §19.13 promised to say where a warehouse credential comes from; that is §28.9.

28.1 What Reproducibility Buys

The claim is that infrastructure defined in files can be recreated, reviewed, and reasoned about. All three are true and they are worth different amounts, which is the part worth being deliberate about.

Recreation is worth the least, and it is the one always cited. Most teams recreate their platform approximately never; the disaster-recovery scenario that justifies it is real and rare, and a plan that has never been exercised is Chapter 19 Case Study 2's hypothesis in another form.

Review is worth the most, and it is barely mentioned in the literature. A change to a warehouse size, an IAM policy, or a bucket lifecycle rule becomes a diff that a second person reads — and Chapter 27 §27.10's argument applies unchanged: the value is that a decision becomes visible at the moment it is made.

Reasoning is worth a surprising amount. "What exists?" and "who can read this bucket?" are questions with an answer in a repository and no answer in a console.

📐 Design Decision — how much infrastructure as code is enough

The maximalist position — everything, including the things you touch once a year — is expensive and is usually argued for on the recreation benefit, which is the weakest one.

A more defensible ordering, by what each buys:

Manage as code	Because
IAM, roles, grants	review — the highest-consequence, least-visible changes
buckets and lifecycle rules	review; a lifecycle rule deletes data silently (Ch. 9)
warehouses and their sizes	review and cost; they change often
the orchestrator's deployment	recreation genuinely matters here
networks, VPCs, subnets	reasoning; changed rarely, catastrophic when wrong
dashboards, BI tool config	usually not worth it
individual tables	no — §28.3

The bottom two rows are where teams over-invest. Encoding a BI tool's configuration in Terraform is possible, is a great deal of work, and buys review of changes that are cheap to reverse — which is the opposite of the ordering above.

The test: what does a mistake in this cost, and how would you find out? IAM scores highest on both. A dashboard's layout scores lowest on both. Spend the effort where the two align.

28.2 Terraform: State, Plan, Apply

Three concepts, and the first is where the difficulty is.

State is Terraform's record of what it believes exists, mapping your configuration to real resource identifiers. It is not derived from reality; it is a file, and when it disagrees with reality you have drift (§28.10).

Plan compares configuration to state to reality, and prints what it would do.

Apply does it.

terraform plan -out=tfplan          # always to a file
terraform show -json tfplan | jq .  # and read it
terraform apply tfplan              # apply exactly what you reviewed

Planning to a file and applying that file is the discipline that matters, because terraform apply without one re-plans at apply time — and the thing it applies may not be the thing you read.

Three things that go wrong, in order of how much they cost:

The state file becomes the most important file you own. Lose it and Terraform believes nothing exists; it will happily create a second copy of your entire platform. Store it remotely, versioned, with locking — S3 with DynamoDB locking, GCS, or Terraform Cloud — and back it up separately from the thing that backs up everything else.

A plan destroys something. §28.3's ⚠️ callout.

Two people apply at once. State locking prevents it; a team that has disabled locking because it was inconvenient has disabled the thing preventing a corrupted state file.

28.3 The Boundary

Terraform can create a Snowflake table. The provider supports it. It should not.

The line, stated once: Terraform owns what must exist before your pipeline runs. Your pipeline owns what it produces.

Two arguments for the line, and the second is the practical one:

Lifecycle. A table is rebuilt nightly; a warehouse is changed quarterly. Terraform's model — declare the desired state, converge to it — fits things that change rarely and fights things that change constantly.

Ownership at 05:00. When a model is wrong, the person fixing it should not need Terraform, an apply, and a state lock. Chapter 26 §26.8's pre-authorized actions have to be executable by the on-call engineer, and anything Terraform owns is not.

⚠️ Failure Mode — the plan line that drops a database

Terraform's plan output marks changes with symbols, and one of them destroys data:

text + create ~ update in place -/+ destroy and then create replacement ← THIS ONE - destroy

-/+ is a replacement, and for most resources it is routine — a security group, a load balancer rule. For a stateful resource it means "delete the thing containing your data and make a new empty one."

What triggers it is a change to a field the provider marks ForceNew, and the list is not intuitive:

text snowflake_database.name ForceNew ← renaming DROPS and recreates aws_s3_bucket.bucket ForceNew aws_rds_instance.engine_version sometimes ← depends on the direction aws_msk_cluster.kafka_version sometimes

A rename is the dangerous case, because it does not look like a destructive change in a diff — it looks like a rename.

Three controls, and use all three:

• prevent_destroy on anything stateful. It makes the apply fail rather than proceed, and the inconvenience of removing it deliberately is the point:

hcl lifecycle { prevent_destroy = true }

• Fail CI on any delete or replace in the plan, unless the pull request carries an explicit approval label. code/plan_review.py does this.
• Read the plan's resource lines, not its summary. Plan: 2 to add, 1 to change, 1 to destroy is a summary that has appeared above a dropped production database, and the count does not say which.

28.4 Docker: The Image Is the Environment

A container image is the only artifact in this book that reliably reproduces an environment, and the reason is that it contains the environment rather than describing it.

Three pinning levels, and most teams stop one short of the useful one:

FROM python:3.11                  # ❌ moves weekly
FROM python:3.11.9-slim           # ⚠️ better; the tag can still be re-pushed
FROM python:3.11.9-slim@sha256:e7f... # ✅ a digest. Immutable, always.

A tag is a pointer and can be moved. A digest is content-addressed and cannot. If a build from last March must produce the same image today, it is digests or nothing.

And the same applies inside the image:

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # pinned ==, not >=

requirements.txt with >= is not a pinned environment, which is why this book's ships with == on all 48 lines and why Chapter 22's benchmark reports the versions it measured on.

Multi-stage builds matter more for data images than for application ones, because the build dependencies are large:

FROM python:3.11.9-slim@sha256:... AS build
RUN apt-get update && apt-get install -y --no-install-recommends gcc g++
COPY requirements.txt .
RUN pip wheel --wheel-dir /wheels -r requirements.txt

FROM python:3.11.9-slim@sha256:... AS runtime
COPY --from=build /wheels /wheels
RUN pip install --no-index --find-links=/wheels /wheels/* && rm -rf /wheels
COPY . /app

Kestrel's image went from 1.8 GB to 410 MB this way, which matters because it is pulled by every Kubernetes task pod — a 1.4 GB saving multiplied by every task start.

💸 Cost Check — the image nobody measured

With KubernetesExecutor, each task runs in its own pod, and each pod pulls the image unless it is cached on the node.

Kestrel runs about 200 task instances a day across eleven DAGs. Before the multi-stage rebuild, on a node pool that scales down overnight and therefore has a cold cache most mornings:

$$200\ \text{pulls/day} \times 1.8\ \text{GB} = 360\ \text{GB/day of image pulls}$$

Within the same region and registry the transfer is free, so the money was small — the time was not. A 1.8 GB pull on a cold node adds 40–70 seconds to task startup, and at 200 tasks a day with roughly 60 cold starts:

$$60 \times 55\ \text{s} = 55\ \text{minutes a night of pulling}$$

Which came directly out of Chapter 25's margin. After the rebuild, 410 MB and about 13 seconds: 13 minutes, a saving of 42.

The lesson is not about Docker. It is that image size is a latency cost paid per task, and it is invisible in every dashboard because it happens before the task starts — so it is attributed to the scheduler, or to "Kubernetes being slow," or to nothing.

28.5 Kubernetes, Honestly

Most data teams do not need Kubernetes, and the ones that have it usually acquired it because the rest of the company had it.

Three genuine reasons, and all three are about isolation rather than scale:

Per-task resource isolation. KubernetesExecutor gives each Airflow task its own pod with its own memory limit — which is Chapter 22 Case Study 1's exit code 137 becoming a contained failure rather than one that takes a worker down.

Conflicting dependencies. One DAG needs pandas 1.5 and another needs 3.0. Per-task images solve this and nothing else does.

It is already there. The marginal cost of one more workload on an existing, operated cluster is genuinely low, and this is the most common good reason.

What it costs, stated plainly because the writing on this rarely does: a cluster to upgrade, a network policy to maintain, autoscaling to tune, pod scheduling failures to debug (Chapter 24 §24.12's "stuck in queued"), and a new vocabulary for every on-call engineer. For a four-person team (Chapter 26 §26.1), that is a substantial fraction of a person.

The alternatives, in order of decreasing operational load:

The last row is not a joke. A LocalExecutor on a well-sized instance runs Kestrel's eleven DAGs comfortably, and the honest reason to move off it is §28.5's first two rows — not throughput.

28.6 Environments From One Module

Three environments should be three sets of variables, not three directories that have diverged.

# modules/warehouse/main.tf -- one definition
resource "snowflake_warehouse" "this" {
  name           = upper("${var.env}_transform_wh")
  warehouse_size = var.warehouse_size
  auto_suspend   = var.auto_suspend
  lifecycle { prevent_destroy = true }
}

# envs/prod/main.tf
module "warehouse" {
  source         = "../../modules/warehouse"
  env            = "prod"
  warehouse_size = "MEDIUM"
  auto_suspend   = 60
}

# envs/ci/main.tf
module "warehouse" {
  source         = "../../modules/warehouse"
  env            = "ci"
  warehouse_size = "XSMALL"     # the only real difference
  auto_suspend   = 30
}

The value is that the difference between environments is a short list you can read — and that a change to the module applies everywhere, so environments cannot drift apart through neglect.

auto_suspend deserves its own note, because it is the highest-leverage line in a Snowflake configuration and is frequently left at the default:

auto_suspend = 600   (10 min)   idle cost after a 40-second query: 10 min
auto_suspend = 60    (1 min)                                        1 min

For a warehouse serving bursty analyst queries, that is most of the bill, and it is one integer.

28.7 Adopting This on a Platform That Already Exists

Almost nobody starts greenfield. The realistic situation is a platform built in consoles over three years by people some of whom have left, and the advice in most infrastructure-as-code writing — which assumes you are creating things — does not address it.

The wrong approach is the one everyone tries first: write the Terraform, run apply, and discover it wants to create a second copy of everything you already have. Terraform's state does not know your platform exists, so every existing resource is, to it, a resource to create.

import is the mechanism, and since Terraform 1.5 it is declarative and reviewable rather than a side-effecting command:

import {
  to = snowflake_warehouse.transform
  id = "TRANSFORM_WH"
}

resource "snowflake_warehouse" "transform" {
  name           = "TRANSFORM_WH"
  warehouse_size = "MEDIUM"
  auto_suspend   = 60
  lifecycle { prevent_destroy = true }
}

terraform plan then shows what it would change to make reality match your configuration — and that plan is the most valuable artifact in the whole exercise, because it is a list of everything about the resource that you got wrong, or that somebody changed and did not tell you.

A strategy that works, in the order that keeps the risk low:

1. Import read-only first, and change nothing. Write configuration matching reality exactly, until plan reports no changes. This is the phase that takes the time, and it is where you discover the platform's actual state.

2. Start with the least dangerous resources. IAM policies and buckets before databases. A mistaken plan on a policy is recoverable; on a database it is §28.3's ⚠️ callout.

3. One resource type per pull request. Reviewable, and a mistake affects one kind of thing.

4. Leave the riskiest resources for last, or never. A production database that has existed for four years, that nobody will ever recreate, and whose accidental replacement is catastrophic is a reasonable thing to leave out — with a comment saying so, which is the difference between a decision and an omission.

🔎 Read the Plan — the import plan is an audit, and it is the point

Kestrel imported 94 existing resources over six weeks. The first plan after each import was the deliverable, and across the 94 it found:

text resources whose config matched reality first time 41 differences the team knew about 22 differences NOBODY KNEW ABOUT 31 ←

Thirty-one. Including:

• A bucket with versioning disabled that everyone believed was versioned — which is Chapter 10's time-travel recovery not existing.
• Two IAM roles with s3:* on *, created during an incident in 2024 and never narrowed.
• A warehouse at LARGE that the team believed was MEDIUM. Nobody could say when it changed; Chapter 25's cost attribution had been reporting the cost correctly and nobody had traced it to a size.
• A lifecycle rule transitioning objects to Glacier after 30 days on a prefix that a monthly job reads — so that job had been paying restore fees, quietly, for over a year.

None of these was found by the import. They were found by writing down what you believe and comparing it to what is there, and the import was merely the thing that forced it.

Which is the reusable idea: the value of adopting infrastructure as code is mostly extracted before you manage anything. If you never got past step 1, the exercise would still have been worth it.

28.8 Splitting State, and Blast Radius

One state file for everything is simple and is a single blast radius. A mistake in a plan can destroy anything in it; a lock held by a long apply blocks every other change; and the plan itself gets slow enough that people stop reading it.

Splitting is the answer and the split is a design decision, not a convention:

infra/
  network/       changes yearly     blast radius: everything
  data-stores/   changes quarterly  blast radius: the data     ← isolate this
  compute/       changes weekly     blast radius: jobs
  observability/ changes weekly     blast radius: monitoring

Two principles for where to cut, and they usually agree:

By rate of change. Things changed weekly should not share a state file — or a lock — with things changed yearly.

By blast radius. Anything whose destruction loses data goes in its own state, applied by a smaller set of people, with prevent_destroy throughout.

What splitting costs is the part usually omitted: cross-state references. Compute needs the network's subnet IDs and the data store's endpoints, and there are three ways to get them, in increasing order of coupling:

# 1. Data source. Loose, and it queries the provider at plan time.
data "aws_vpc" "main" { tags = { Name = "kestrel-prod" } }

# 2. A published output, read from the other state. Explicit, and it
#    creates a hard dependency between the two states' apply order.
data "terraform_remote_state" "network" {
  backend = "s3"
  config  = { bucket = "kestrel-tfstate", key = "network/terraform.tfstate" }
}

# 3. A variable, set by hand. Ugly, decoupled, and honest.
variable "subnet_ids" { type = list(string) }

Prefer 1, accept 2, and use 3 for the boundary you most want to keep loose — which is usually the one crossing an ownership line.

📐 Design Decision — how many state files is right

Kestrel started with one and moved to four, and the reasoning was not the usual one.

The trigger was not size. The plan was fast enough. The trigger was §28.3's ownership argument arriving in practice: an on-call engineer needed to change a compute resource at 05:30, and the apply would have planned changes to the network and the data stores as well — including a drifted warehouse size (§28.10) that had nothing to do with the incident.

A state file's real unit is not "a set of resources." It is "a set of changes that must be applied together by the same person." That framing makes the split obvious, and it explains why splitting by environment is universal while splitting by layer is contested: environments genuinely are applied separately, and layers only sometimes are.

What Kestrel would do differently: start with two — stateful and stateless — from the first day. The split that matters is the one protecting data, and retrofitting it required moving resources between states, which is the one Terraform operation with no good story.

And a warning against the other extreme. A state file per resource is technically possible and produces a platform where nothing can be reasoned about, every change is six pull requests, and the cross-state references outnumber the resources. Four is a good number for a small platform; forty is a different problem.

28.9 Where a Credential Comes From

Chapter 19 §19.13 said env_var() and pointed here. The end-to-end path, with no long-lived secret anywhere:

 GitHub Actions                  Cloud IAM                    Warehouse
 ─────────────                   ─────────                    ─────────
 job requests an                 validates the OIDC
 OIDC token         ──────────▶  token's claims        ──┐
 (id-token: write)               (repo, ref, env)         │
                                                          ▼
                                                   issues short-lived
                                                   credentials (1 hour)
                                                          │
                                                          ▼
                                              reads the warehouse secret
                                              from Secrets Manager
                                                          │
                    dbt reads DBT_PASSWORD ◀───────────────┘
                    from the environment

Four properties, and the third is the one that changes incident response:

No long-lived credential exists in the repository, in CI configuration, or on a laptop.

The token is scoped to a repository and a branch. A fork's pull request cannot assume the role.

Rotation is not an event. Credentials last an hour and are reissued; there is no rotation runbook, because there is nothing to rotate — which removes an entire category of the Chapter 26 runbooks that go stale.

Every use is audited, with the workflow run that requested it.

For the human path, the same idea: SSO to an identity provider, short-lived warehouse tokens, and no shared account. The failure mode a shared service account produces is Chapter 22 Case Study 2's "read by a service account nobody could identify," and it is unanswerable after the fact.

🔐 Privacy & Governance — the three credentials a data platform forgets

The warehouse password gets attention. These three do not, and each has been a real breach somewhere:

The Airflow metadata database. It contains connection details for every system you integrate with, and Chapter 24 Case Study 2 established that it also frequently contains XCom values — which may be actual data. It is a high-value target treated as plumbing.

The BI tool's warehouse connection. Usually a service account, usually broadly privileged because narrowing it broke a dashboard once, and usually the oldest credential in the system.

A developer's local profiles.yml. Chapter 19 §19.13. It is on a laptop, it is not rotated, and it frequently has production access because deferring to production requires it (Chapter 27 §27.4).

The audit that finds these is one question: what can read production, and how long has each of those credentials existed? Kestrel's first pass found eleven identities with production read access, of which four were unattributable and two belonged to people who had left.

And the fix for the last category is not rotation. It is short-lived credentials issued against an identity, so that "who left" and "what can read production" are the same question answered in one place.

28.10 Drift Is Inevitable

Drift is when reality and state disagree, and the literature treats it as a discipline failure. It is not, and treating it that way produces a worse platform.

Chapter 26 §26.8 pre-authorizes the on-call engineer to act. At 05:20, with the SLA in forty minutes, resizing a warehouse in the console is the correct action — and it creates drift.

So the design goal is not zero drift. It is:

1. Detect it, quickly and automatically.
2. Reconcile it deliberately — either the change is right and belongs in code, or it was temporary and should be reverted.
3. Never surprise anyone with it, which means never letting an unrelated apply silently revert someone's fix.

# Nightly. Exit 2 means drift.
terraform plan -detailed-exitcode -refresh-only

🏭 From the Pipeline — the apply that undid the fix

The failure this section exists to prevent:

text 05:20 on-call resizes TRANSFORM_WH from MEDIUM to LARGE. SLA met at 05:58. 05:59 incident closed. The resize is not mentioned; it was a mitigation. 11:40 an unrelated PR adds a new bucket. terraform apply. 11:40 the plan ALSO reverts the warehouse to MEDIUM. Nobody reads that line. next the same incident, and the same fix, and the same reversion day

The warehouse resize was correct. The Terraform apply was correct. The failure is that the second one silently undid the first, and the person who applied it was not the person who had made the change.

Three things that prevent it, and the first is nearly free:

• A nightly drift check that posts to the channel, naming the resource and the difference. The on-call engineer sees it the next morning and decides, which is the reconciliation step.
• A plan review that flags reversions specifically. A ~ update in place that moves a value back to what the code says, on a resource nobody touched in this pull request, is a reversion and deserves a line of its own.
• A note in the incident record. Chapter 26 §26.9's template gains one field: "did you change anything outside code?"

And the cultural half: drift found by the nightly check is information, not a violation. A team that treats it as a process failure will get engineers who make undocumented changes rather than documented ones, which is strictly worse.

28.11 Cost as Code

Tagging is the only mechanism that makes cloud cost attributable, and it is the thing every team intends to do and does not.

locals {
  tags = {
    team        = "data-engineering"
    environment = var.env
    component   = "warehouse"
    owner       = "data-eng@example.com"
    managed_by  = "terraform"       # ← so an untagged resource is visible
  }
}

managed_by is the useful one, because it makes the console-created resources findable:

# Anything in the account NOT created by Terraform.
aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=managed_by,Values=terraform --query 'ResourceTagMappingList'

Enforce the tags rather than documenting them — a default_tags block on the provider applies them to everything, and a policy check fails the plan on anything missing them.

And budgets belong in the same repository as the thing they constrain:

resource "aws_budgets_budget" "data_platform" {
  budget_type  = "COST"
  limit_amount = "4200"
  cost_filter { name = "TagKeyValue"
                values = ["user:team$data-engineering"] }
  # 80% is a conversation; 100% is a page. Ch. 26 §26.4.
  notification { threshold = 80  ... }
  notification { threshold = 100 ... }
}

Chapter 25 §25.6's per-job cost attribution needs this to exist, and it is the piece most teams discover they are missing when they try to build it.

🧭 Version Note — Terraform 1.5+, and the OpenTofu fork

The mechanics in this chapter have moved recently enough that older material describes a different tool:

• The import block is Terraform 1.5+. Before that, terraform import was a side-effecting command that mutated state directly and could not be reviewed — which is why §28.7's strategy would have been much harder to run in 2022, and why a lot of "adopting IaC" writing recommends greenfield rebuilds instead.
• terraform test is 1.6+, and is a genuine testing framework rather than a third-party harness. §28.12's note about it being worth it for a published module still applies.
• moved blocks (1.1+) let you rename a resource in configuration without a destroy-and-create, which is §28.3's ⚠️ callout's most common trigger, defused. If you are still renaming resources by editing state, this is the thing to adopt.
• check blocks (1.5+) assert post-apply conditions — that a bucket really is versioned, that an endpoint responds — which is §28.12's "assert that the check ran," in Terraform's own vocabulary.

OpenTofu is a fork of Terraform 1.5, created after HashiCorp changed Terraform's licence from MPL to BUSL in 2023. It is under the Linux Foundation, is a drop-in replacement at that version, and has since added features of its own.

What matters for a reader: the concepts, the file format, the state model, and everything in this chapter apply to both, and the choice is a licensing and governance question rather than a technical one. Check which your organization has standardized on before writing a module, because the divergence is small today and growing.

28.12 Testing Infrastructure

Four levels, and the first two cover most of the value:

Static checks. terraform validate, tflint, and a policy checker — checkov, tfsec, or OPA. Seconds, no cloud account, and they catch a public bucket.

Plan review. code/plan_review.py — parse the plan JSON and fail on destroys, replacements, IAM broadening, and lifecycle-rule changes. §28.3's ⚠️ callout, automated.

Ephemeral environments. terraform apply into a throwaway account or project, run assertions, destroy. Expensive and slow, and worth it for a module you are changing rather than for every plan.

Production reconciliation. §28.10's drift check.

The one thing that is not on the list is "unit testing Terraform," which exists (Terratest, and terraform test since 1.6) and is worth it for a module you publish rather than for configuration you apply once.

And one assertion worth adding that is not a test of Terraform at all: after every apply, check that the thing you believe you created actually behaves as expected.

# Post-apply, in CI. Cheap, and it catches the class §28.7's import found.
aws s3api get-bucket-versioning --bucket kestrel-lake   | jq -e '.Status == "Enabled"' || { echo "lake is NOT versioned"; exit 1; }

Terraform reported success; the bucket's versioning is a separate fact. A provider bug, a race with a bucket policy, an eventually-consistent read, or simply a resource argument that does not mean what its name suggests — and Chapter 27 Case Study 2's rule applies here too: assert that the thing happened, not that the command succeeded.

🎓 Interview Angle — "how do you manage your infrastructure?"

A question that invites a tool name and is scored on the boundary you draw.

The strong answer names the boundary first:

"Terraform manages the containers, not the contents — the buckets, the warehouse, the roles, the network — and the pipeline manages tables and partitions. The reason is that a table's schema is defined by the model that populates it, so managing it in Terraform means two sources of truth, and a resource replacement then deletes data. Beyond that: remote state, locked and backed up separately; prevent_destroy on every stateful resource, plus a plan reviewer that fails on a destroy — two controls, because a control you've never seen fail is a hypothesis. And OIDC from CI rather than a long-lived key, with a condition on the subject claim so a fork can't assume the role."

Four things that answer does. It states the boundary and the reason. It names two independent controls rather than one. It mentions the state backup, which almost nobody does and which is the single point of failure. And it gets the OIDC detail right — the condition, not just the protocol.

The follow-ups:

"What happens if someone changes something by hand?" — a nightly drift plan, posted somewhere people read. And the good answer adds the classification step: adopt, revert, or leave with a note — because reverting drift blindly re-breaks whatever it fixed.

"How do you adopt this on a platform that already exists?" — import until plan is empty. And the artifact worth having is the inventory, not the configuration: every team that does this discovers resources nobody knew about.

"Do you use Kubernetes?" — and the strong answer prices it. Roughly twenty engineer-days a year for a small team, against managed alternatives that provide the isolation — a candidate who can say "no, and here is the number" is demonstrating exactly the judgment the question is for.

And a detail worth volunteering: terraform plan succeeding is not the same as the resource behaving. A post-apply assertion — an unauthenticated GET returning 403, a versioning probe that actually versions — is Chapter 23's distinction applied to infrastructure, and very few candidates have thought about it that way.

📏 Scale Note — one state file, and when it stops being one

Terraform state is a single file with a single lock, and every property of your workflow follows from that.

text resources plan time what starts hurting ───────────────────────────────────────────────────────────────────────── < 100 ~20 s nothing 100-300 1-3 min the refresh dominates; `-refresh=false` becomes tempting, and it is how drift hides 300-800 3-10 min the LOCK. Two people cannot apply at once, and a CI apply blocks a human for ten minutes. 800+ 10+ min a plan nobody reads to the end, which is the real failure -- Case Study 1's twelve unprotected resources were IN a plan

The lock row is where teams first feel it and the last row is where the damage is. A ten-minute plan containing four hundred unchanged resources and three changes is a document that gets skimmed, and skimming is how a must be replaced on a stateful resource gets approved.

The split that fixes both is Exercise 28.24's, and the direction matters:

```text infra-stateful/ buckets, databases, backup vaults, KMS keys applied RARELY, by a human, with review prevent_destroy on everything

infra-stateless/ IAM roles, network, compute, monitoring applied by CI, freely, because nothing here can destroy data

direction of dependency: stateless READS stateful. Never the reverse. ```

The benefit is realised on every plan afterwards. The stateless state can be applied automatically because nothing in it can lose data, and the stateful state's plan is short enough to read.

Do it before you need to (Exercise 28.24). Moving resources between states has no dry run, no atomicity, and a failure mode where the resource is in neither state and Terraform proposes to create it — which, for a bucket, means Terraform proposes to create a bucket that already contains your lake.

🔁 Idempotency Check — terraform apply is idempotent and your bootstrap is not

Terraform's whole model is convergence: apply twice, get the same infrastructure. That holds for the resources Terraform manages and stops at every boundary where you reached outside it.

text operation idempotent? ───────────────────────────────────────────────────────────────────────── terraform apply YES, by design a local-exec provisioner running a shell script ONLY IF the script is. It usually is not, and it runs on CREATE only, which hides the problem. a null_resource with a trigger runs again when the trigger changes, which is a different thing from "converges" a bootstrap script that seeds a database almost never. `CREATE TABLE` is; an `INSERT` of reference rows is not. a bucket lifecycle rule applied by a separate CLI call it converges, and Terraform does not know about it -> drift, forever

Row two is the trap with the sharpest edge. A local-exec provisioner runs when the resource is created and never again, so a script that is subtly wrong produces infrastructure that is subtly wrong and a plan that says "no changes." The error is frozen into the resource.

Three rules:

Provisioners are a last resort, and the documentation says so. If a resource needs configuring after creation, prefer a provider that does it, or a separate idempotent job that runs on a schedule and converges.

Any bootstrap must be safe to re-run. CREATE TABLE IF NOT EXISTS, INSERT ... ON CONFLICT DO NOTHING, mc mb --ignore-existing. Exercise 5.19's minio-init is exactly this, and its race condition (Exercise 5.10) is the other half of the same problem.

And the real test is Chapter 5's 🔁, one layer up: destroy it and bring it back. terraform destroy on a non-production copy, then apply, then the post-apply assertions (Exercise 28.18). A stack that has never been rebuilt from its own code is a stack whose code is a description rather than a definition — and the difference is only ever discovered at the worst possible time.

🧪 Try It — try to destroy something, and watch two controls stop you

A control you have never seen fail is a hypothesis (§27's lesson, applied here). This takes fifteen minutes and it is Exercise 28.23(c).

```bash

1. Make a change that FORCES REPLACEMENT of a stateful resource.

On a bucket, changing the name does it. Do this in a non-production

workspace, obviously.

terraform plan should say:

# aws_s3_bucket.bronze must be replaced

-/+ resource "aws_s3_bucket" "bronze" {

2. CONTROL ONE: prevent_destroy

terraform plan

Error: Instance cannot be destroyed

Resource aws_s3_bucket.bronze has lifecycle.prevent_destroy set

3. Now REMOVE prevent_destroy in the same change -- which is what

somebody in a hurry would do -- and plan again.

terraform plan -out=tf.plan

...it plans successfully. Control one is bypassed.

4. CONTROL TWO: the plan reviewer

terraform show -json tf.plan | python plan_review.py

BLOCKED: aws_s3_bucket.bronze -> ["delete","create"]

and the resource carries tag managed_by=stateful

```

Step 3 is the point of the exercise. prevent_destroy is a strong control and it lives in the same file as the change it is protecting against, so it can be removed by the same commit. A reviewer looking at a large diff will not notice three deleted lines.

Control two lives outside the configuration — it reads the plan's JSON and it does not care what the configuration says. That independence is the property, and it is the reason for two controls rather than a better one.

Record both refusals verbatim in docs/known-errors.md, and then do the thing people skip: check that the reviewer would have caught it if you had not removed prevent_destroy either. Two controls that both fail on the same input have been tested once.

🔐 Privacy & Governance — the state file is a map of everything you have

Terraform state is not configuration. It is a JSON document containing the current attributes of every managed resource, and several of those attributes are secrets.

text what is in a state file ───────────────────────────────────────────────────────────────────── every resource id, arn, endpoint, and name a complete inventory initial database passwords, if a resource generated one in PLAINTEXT generated keys and tokens in plaintext the outputs, including anything marked sensitive redacted in the CLI, present in the FILE the full attribute set of every resource which is a much better reconnaissance document than the configuration

sensitive = true redacts a value from the plan output and does not remove it from the state. That is documented and it surprises nearly everyone, and it means the state file is at least as sensitive as the most sensitive thing Terraform manages.

Four controls, all of them configuration:

Remote state, encrypted at rest, with versioning and a lock. Never local, never in git — a state file committed once is a secret in the history forever.

A separate, tighter access policy on the state bucket than on the resources it describes. Read access to state is functionally read access to every credential Terraform generated.

A separate backup, on a different schedule from everything else (Exercise 28.23a). State is the one object whose loss cannot be recovered from the resources themselves — Terraform would propose to create everything you already have.

And prefer generating secrets outside Terraform. A password created by a secrets manager and referenced by Terraform never enters state; one created by random_password does. That is a one-line design difference with a permanent consequence.

The observation worth carrying: infrastructure as code moves your infrastructure's description into a repository, which is exactly what you wanted — and it moves your infrastructure's state into a file that nobody thinks of as data.

🧭 Version Note — the parts of this chapter with the shortest half-life

text stable for years changes constantly ───────────────────────────────────────────────────────────────────── the boundary (containers, not provider resource names and contents) arguments remote state, locked, versioned the exact backend configuration prevent_destroy module registry conventions plan / apply / import provider version constraints OIDC + a subject-claim condition the claim's exact format, which differs per CI system

Two specific things worth pinning rather than remembering.

Provider versions. ~> 5.0 is not a pin; it is a range, and a minor provider release can change a default that produces a plan you did not ask for. Use a lock file and commit it — .terraform.lock.hcl exists for this and is frequently gitignored by accident.

And the OIDC subject-claim format. GitHub's is repo:owner/name:environment:prod; GitLab's, Bitbucket's, and a self-hosted runner's are all different, and a trust policy that matches too loosely accepts a token it should not. Check the exact string against a real token rather than against a blog post.

What does not change is the reason for any of it, and it is worth restating: infrastructure as code exists so that a change is reviewable before it happens. Every other benefit — recovery, consistency, drift detection — follows from having a diff. A workflow that applies without a reviewed plan has the tool and not the property.

28.13 The Kestrel Platform

🧱 Kestrel Platform — Increment 28: the platform, in files

text infra/ modules/ warehouse/ snowflake warehouse, roles, grants lake/ buckets, lifecycle (Ch. 9), encryption orchestration/ the Airflow deployment observability/ budgets, alerts, the metrics stack envs/ prod/ ci/ dev/ variables only -- §28.6 policies/ no_public_buckets.rego prevent_destroy.rego docker/ Dockerfile multi-stage, digest-pinned .github/workflows/ infra.yml validate → policy → plan → plan_review → apply drift.yml nightly, posts to the channel

Eight things this increment must get right:

1. Terraform state is remote, versioned, locked, and backed up separately. §28.2.
2. The boundary is respected: no snowflake_table resources anywhere. §28.3.
3. prevent_destroy on every stateful resource, and plan_review.py fails CI on any destroy or replacement without an approval label.
4. The image is digest-pinned and multi-stage. Record the size before and after.
5. Three environments, one module, and the differences are a readable list. §28.6.
6. No long-lived credentials: OIDC to a role, short-lived warehouse tokens, secrets in a manager. §28.9.
7. A nightly drift check that posts to the channel — and drift is treated as information. §28.10.
8. default_tags with managed_by, and a budget in the same repository as the thing it constrains.

The exercise that matters is 28.23(c): write a plan that would replace a stateful resource, and confirm that both prevent_destroy and plan_review.py stop it. Two independent controls, each tested — because Chapter 27 Case Study 2's lesson is that a control you have not seen fail is a hypothesis.

28.14 Summary

Reproducibility buys recreation, review, and reasoning — and review is worth the most while being the one least often cited. 📐 Manage as code where a mistake is expensive and hard to notice: IAM first, bucket lifecycle second, warehouses third. Dashboards and individual tables, no.

Terraform's state is a file, not a derivation. Remote, versioned, locked, backed up separately. Plan to a file and apply that file, because apply without one re-plans and may not do what you read.

The boundary: Terraform owns what must exist before your pipeline runs; the pipeline owns what it produces. Two reasons — lifecycle (Terraform fits things that change rarely) and ownership at 05:00 (Chapter 26's pre-authorized actions must not require an apply and a state lock).

⚠️ -/+ is a replacement, and on a stateful resource it means "delete the thing with your data." A rename is the dangerous case, because in a diff it looks like a rename. prevent_destroy, a CI check on the plan JSON, and read the resource lines rather than the summary — Plan: 2 to add, 1 to change, 1 to destroy has appeared above a dropped production database.

Pin images by digest, not tag. A tag is a pointer and can be moved. 💸 Multi-stage builds took Kestrel's image from 1.8 GB to 410 MB, and image size is a latency cost paid per task — 55 minutes a night of pulling became 13, straight out of Chapter 25's margin, and invisible on every dashboard because it happens before the task starts.

Kubernetes: three genuine reasons, all about isolation — per-task memory limits, conflicting dependencies, and an existing cluster. Not scale. The cost is a substantial fraction of a person on a four-person team, and LocalExecutor on one well-sized machine covers more teams than admit it.

Three environments, one module, differences as a readable list. And auto_suspend is the highest-leverage integer in a Snowflake configuration.

🔎 Adopt this on an existing platform by importing read-only first, changing nothing until plan is clean. The first plan after each import is the deliverable: Kestrel's 94 imports found 31 differences nobody knew about — a bucket believed versioned and not, two roles with s3:* on *, a warehouse at LARGE, and a lifecycle rule sending a monthly job's data to Glacier. The value is extracted before you manage anything.

📐 A state file's real unit is "a set of changes that must be applied together by the same person." Split stateful from stateless on day one — retrofitting it means moving resources between states, which is the one Terraform operation with no good story. Four state files is a good number for a small platform; forty is a different problem.

🔐 A credential should not exist. OIDC → a short-lived role → a secret manager → an environment variable. Rotation stops being an event because there is nothing to rotate. The three forgotten credentials: the Airflow metadata database (it holds connections, and often XComs), the BI tool's service account (broad, and the oldest in the system), and a developer's profiles.yml.

🏭 Drift is inevitable and is not a discipline failure. Chapter 26 pre-authorizes the 05:20 console change, and that change is correct. Detect it nightly, reconcile it deliberately, and never let an unrelated apply silently revert someone's fix. A team that treats drift as a violation gets undocumented changes instead of documented ones.

Tag everything, including managed_by, so console-created resources are findable — and put the budget in the same repository as the thing it constrains, because Chapter 25's per-job cost attribution needs it to exist.

Test infrastructure in four levels, and the first two — static checks and plan review — cover most of the value in seconds, with no cloud account.

Part VI opens with Chapter 29 and streaming architecture: the design decisions that follow once some of this has to happen in seconds rather than overnight.

Key terms: declarative · Terraform state · plan · apply · import · remote state · blast radius · ForceNew · prevent_destroy · module · digest pinning · multi-stage build · KubernetesExecutor · drift · refresh-only plan · OIDC · workload identity · default_tags · managed_by · budget