Chapter 28 — Key Takeaways (Infrastructure as Code)

The page to read before your first terraform apply against production, and again before renaming anything.

What it buys

Recreation · review · reasoning — and 📐 review is worth the most while being the one least cited. Most teams recreate their platform approximately never.

Manage as code where a mistake is expensive and hard to notice:

| IAM, roles, grants | highest on both | | buckets and lifecycle rules | a lifecycle rule deletes data silently | | warehouses and sizes | review and cost; they change often | | the orchestrator's deployment | recreation genuinely matters here | | BI config, individual tables | no |

Terraform

State is a file, not a derivation. Remote, versioned, locked, and backed up separately from whatever backs up everything else. Lose it and Terraform will build a second copy of your platform.

Plan to a file and apply that file — apply without one re-plans and may not do what you read.

The boundary

Terraform owns what must exist before your pipeline runs. Your pipeline owns what it produces.

Two reasons: lifecycle (Terraform fits things that change rarely) and ownership at 05:00 — Chapter 26's pre-authorized actions must not require an apply and a state lock.

The destructive plan

  + create      ~ update in place      -/+ DESTROY AND RECREATE      - destroy

⚠️ -/+ on a stateful resource means "delete the thing containing your data." It is forced by a ForceNew attribute, and a rename is the dangerous case, because in a diff it looks like a rename.

Plan: 2 to add, 1 to change, 1 to destroy is a summary. The count does not say which — and most plans contain a routine destroy, so by the time one matters the count carries no information.

Two independent controls, failing differently: - prevent_destroy — the apply fails; removing the block is a visible line - A CI check on the plan JSON — the resource is named, above the fold

Images

Pin by digest, not tag. A tag is a pointer and can be moved; a digest is content-addressed. requirements.txt with >= is not a pinned environment either.

💸 Multi-stage: 1.8 GB → 410 MB. Image size is a latency cost paid per task — 55 minutes a night of pulling became 13, straight out of Chapter 25's margin, and invisible on every dashboard because it happens before the task starts.

Kubernetes, honestly

Three genuine reasons, all about isolation: per-task memory limits · conflicting dependencies · it is already operated (the most common good one). Scale is not on the list.

Cost: a cluster, network policy, autoscaling, pod-scheduling debugging, and a new vocabulary for every on-call engineer — a substantial fraction of a person on a team of four. LocalExecutor on one well-sized machine covers more teams than admit it.

Adopting it on a platform that exists

Import read-only first and change nothing until plan is clean. Least dangerous resources first; one type per pull request; leave the riskiest out, with a comment saying so.

🔎 The first plan after each import is the deliverable. Kestrel's 94 imports found 41 matching, 22 known differences, and 31 nobody knew about — a bucket believed versioned and not, two roles with s3:* on *, a warehouse at the wrong size, a lifecycle rule sending a monthly job's data to Glacier. The value is extracted before you manage anything.

Splitting state

📐 A state file's real unit is "a set of changes that must be applied together by the same person."

Split stateful from stateless on day one. Retrofitting means moving resources between states — the one Terraform operation with no good story. Four files is good for a small platform; forty is a different problem.

Cross-state references, loosest first: a data source · terraform_remote_state · a hand-set variable, for the boundary you most want to keep loose.

Credentials

🔐 A credential should not exist. OIDC → a short-lived role → a secret manager → an environment variable. Rotation stops being an event because there is nothing to rotate.

Three a data platform forgets: the Airflow metadata database (every connection, often XComs — a high-value target treated as plumbing) · the BI tool's service account (broad, and the oldest) · a developer's profiles.yml (unrotated, often with production access).

The audit is one question: what can read production, and how long has each of those existed? Kestrel found eleven identities, four unattributable, two belonging to people who had left.

Drift

🏭 Drift is inevitable and is not a discipline failure. Chapter 26 pre-authorizes the 05:20 console resize, and that change is correct.

Three goals: detect nightly · reconcile deliberately · never let an unrelated apply silently revert someone's fix.

Convergence without notification is the defect. The general form appears everywhere: a config-management tool reverting a hotfix, a --full-refresh reverting a manual patch, an auto-scaler reverting a scale-up, a deploy reverting a debug flag. In every case the second actor is doing its job.

Post the check after the on-call window, and word it as information — "if this reads as an accusation, people will make undocumented changes instead of documented ones."

Cost and testing

default_tags including managed_by, so console-created resources are findable by their absence. Put the budget in the same repository as the thing it constrains — Chapter 25 §25.6's per-job cost attribution needs it to exist.

Four levels: static checks · plan review · ephemeral environments · drift. The first two cover most of the value in seconds with no cloud account.

And assert that the thing behaves, not that the command succeeded — a bucket's versioning is a separate fact from a successful apply.

The two case studies

Two to add, one to change, one to destroy. A four-line rename dropped the production analytics database in nine seconds. Apply complete! — because Terraform did exactly what it was asked. The first signal was a broken dashboard nineteen minutes later, because nothing in a nightly-cadence monitoring system fires at 14:12 on a Wednesday.

🔎 A restore returns the object, not its context. UNDROP took four seconds; restoring the grants took ninety minutes, most of it debugging permission errors reported as connection failures.

Set retention from the failure it protects against. Seven days for a destruction detected in minutes; Chapter 24 Case Study 1's slow corruption needed 141 — Time Travel is the right control for a destruction and the wrong one for a slow corruption.

The fix that kept getting undone. Seven correct on-call resizes, seven silent reversions by unrelated applies, over six months. 31.5 hours of repeated diagnosis, because seven postmortems each concluded the same thing and none referenced the previous.

"If this is the second occurrence, the action item is not the same mitigation." A mitigation applied twice is a process; automate it or eliminate it.

📐 $7,592 a year took eleven weeks to approve; 31.5 invisible engineering hours were spent without anyone approving anything. A budget line requires a decision and on-call time does not — pricing the mitigation is what moved it. And a capacity increase is the fastest fix and the one to be least satisfied with: the margin work recovered 168 minutes for nothing per year.