Chapter 27 — Key Takeaways (CI/CD for Data Pipelines)

The page to read before designing a pipeline, and again before merging a definitional change.

The claim

A data deploy has two halves and most teams build only one. Shipping the code is genuinely just software. Getting the data into the state the new code implies — a backfill, a migration, a rebuild — is unique to this work, cannot be rolled back, and is usually left to whoever notices.

Four ways this is harder

  1. You cannot easily test against realistic data — production is where the pathologies live
  2. The build is slow and expensive — 22 min and $2.93
  3. There is no rollback for data — git revert is four minutes and does nothing to rows
  4. Staging is not representative in the one dimension that matters

📐 They have no common fix, and three of the four resolve to "accept something." Data CI/CD is a set of trades. A team that has not named what it gave up gave up something anyway — and the one thing not to trade is speed, because a slow pipeline gets routed around.

Order by yield

1. lint / format          3 s
2. parse                  8 s      ← Ch. 24 §24.2's parse time
3. POLICY LINTS          12 s      ← wall clock, hardcoded refs, uncast money
4. unit tests            25 s
5. slim build           3.1 min
6. full build          22.0 min    ← nightly only

Stages 1–4 need no warehouse. 🔎 Kestrel's policy lints caught 14 failures at 12 seconds each; the 4 caught only by the nightly build were all things slim CI structurally cannot see.

Test data: all three, and none is enough

| Fixtures | logic | ✗ shape, cardinality, the real nulls | | Sampled production | shape | ✗ the rare row | | Synthetic | known pathologies | ✗ the one you did not think of |

⚠️ Random sampling preserves the distribution and destroys the tail. A 1% sample contains one of the twelve wholesale distributors with probability $1-0.99^{12} = 11.4\%$.

Sample deliberately: unusual entities in full · the extremes of every measure · every row that has ever failed a test ← build this clause first · then random, for the shape.

And an integration test, nightly, on synthetic data — small, and it exercises the wiring, which breaks on the changes that look safest.

Slim CI

Three requirements: somewhere to keep the production manifest · a production read grant (a governance decision, not a task) · and a fallback to a full build when it is missing.

⚠️ Cannot see: a hardcoded reference (Ch. 19 CS1 — the check that would find it is the one it disables) · a shared target · some macro changes · anything outside dbt. It is a latency optimization, not a coverage one.

Environments

Staging has no clean answer — a small sample misses the row, a full copy is production with the same PII, synthetic exercises what you thought of. The industry replaced staging with a permission grant, which is what --defer is.

Kestrel deleted theirs: eighteen months, a 5% sample, never once caught something the slim build did not, $340 a month.

Versioning

Code (git) · schema (forward-only migrations) · and for anything in an external report, the VALUE. A table of (metric, period, value, computed_at, code_version) is kilobytes a year and is the only thing that answers "what did we say, and when?" — because a rebuild destroys the number a filing was made on.

Deploy shape

| Additive | none — the new column is null for history | | Definitional | full rebuild, or history is inconsistent | | Structural | migration + rebuild |

🔁 Require it in the pull request, and for the latter two the rebuild command with a dry run. An author who cannot write that command has not thought about the data half.

Three options when you fix a definition, and exactly one is "do nothing": rebuild · do not rebuild and record the boundary · do not rebuild and do not record it. The third is the absence of a decision, and it is the default because shipping the code is the half with a button.

⚠️ A correct fix that leaves the table inconsistent is worse than the bug. A uniformly wrong column can be corrected; one whose definition changes at an unrecorded date cannot.

🔎 Put _built_by (the git SHA) on the row. 40 bytes, a few kilobytes dictionary-encoded, and it turns "which code produced this?" from four days into one query. It is not for the analyst.

Rollback

| Code | git revert. Minutes. | | Table contents | time travel, inside retention | | A migration | a forward migration; the reverse is often lossy | | Data sent downstream | nothing |

Make the irreversible step last, after the assertions: build >> test >> reconcile >> export.

Where rollback is unavailable, run both and compare. Shadow-build and diff — expect differences and read them; the gate is "only the ones you intended." Blue-green swap keeps the old table a day.

Reviewing

📐 If a defect is invisible in the artifact under review, no amount of diligence is the fix.

Delegate to a linter Keep for a human
hardcoded refs, wall clock, uncast money, missing max_active_runs, coverage the deploy shape · is the definition right · did the grain change · who downstream should be told

After moving the left column into CI, Kestrel's review comments fell from a median of 4 to 2 and the share about meaning rather than mechanics rose from a quarter to three quarters.

The orchestrator

⚠️ A DAG deploy affects everyone — the scheduler parses every file every 30 seconds. Parse time is a gate, not a test. Sync atomically. A task rename is a structural change to a historical record: keep task_id stable and rename the function.

Secrets and speed

Never in the repo, never in profiles.yml, prefer OIDC, and --store-failures so a failing test does not print production rows into a log everyone can read.

Four levers on speed: slim CI (22 → 3.1 min) · order by yield · cache dbt deps · parallelize the warehouse-free half.

💸 Put the consequence where the decision is made. A six-line PR comment — blast radius, exposures, deploy shape, margin impact — has never blocked a change, and made four authors find a cheaper approach and three ask a stakeholder before merging, which had never happened and was not requested.

The two case studies

The definition that changed on a Tuesday. A correct three-line fix, two reviewers, all tests green — and net_revenue_cents meant two different things either side of a deploy timestamp nobody chose. 136,664 lines differed; order totals were right throughout, so nothing reconciled differently. Classify conservatively: a false definitional costs thirty seconds, a false additive cost eleven weeks — and measure the override rate, because it is the justification.

The green build that tested nothing. A missing manifest made state:modified+ select nothing, dbt exit 0, and CI pass in eleven seconds for nine weeks and 63 pull requests.

⚠️ A check whose failure is indistinguishable from having nothing to check will fail open — an empty suite passes, PASS=0 TOTAL=0 exits 0. Assert that the check RAN, not that it passed.

And it got faster when it broke, which is why it survived nine weeks: a control that gets slower when it breaks is investigated in a day.

🔎 "Do not X; instead Y" is two requirements, and the negative one is easier, is what a reviewer sees, and is the one implemented alone. Write the positive clause first and test the negative path.

Prefer a loud degraded mode to a blocking one. A blocked person routes around it; an inconvenienced person complains, and a complaint is a detection.