Further Reading: CI/CD for Data Pipelines

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

About seventy percent of this topic is ordinary software delivery, and that seventy percent has excellent books. Read them. The remaining thirty — testing without production data, the unrollbackable half of a deploy, and staging that cannot be representative — has almost no literature, which is why §27.1's four differences are stated as a list rather than cited.

The seventy percent

• Jez Humble and David Farley, Continuous Delivery (Addison-Wesley, 2010). Old, foundational, and still the best statement of the argument. Read the deployment-pipeline chapter and the chapter on managing data, the latter being the one place in the mainstream CD literature that takes database state seriously — and note that even there, the problem it addresses is schema migration rather than the definitional-change problem in Case Study 1. Tier 1.

• Nicole Forsgren, Jez Humble, and Gene Kim, Accelerate (IT Revolution, 2018). The evidence base: deployment frequency, lead time, change failure rate, and time to restore. The relevant transfer is the finding that speed and stability are correlated rather than traded, which is §27.13's argument with data behind it. What does not transfer cleanly is "time to restore," because §27.8's fourth category has no restore at all. Tier 1.

• Michael Nygard, Release It! (2nd ed., Pragmatic Bookshelf, 2018). On deploying things that cannot simply be restarted. The material on stability patterns and on making irreversible operations late and explicit is what §27.8's ordering rule comes from. Tier 1.

• The GitHub Actions, GitLab CI, or Buildkite documentation for whichever you use — specifically the pages on artifacts, caching, and OIDC. §27.4's manifest, §27.13's caching, and §27.12's keyless auth are all one documentation page each, and Case Study 2 is what happens when the artifact page is skimmed. Tier 1.

The thirty percent

• The dbt documentation on state comparison, --defer, and artifacts. The primary source for §27.4, and the section on what state:modified does and does not detect is worth reading against §27.4's ⚠️ callout — the documentation is accurate and the limitations are easy to miss because they are stated as capabilities. Tier 1 — versioned; read the 1.9+ page.

• The dbt "Best Practices" guide on CI/CD, and the community writing around slim CI. Read it for the mechanics and supply your own fallback, because Case Study 2's failure — a missing manifest selecting nothing and passing — is not something the guides warn about. Tier 2 — evolving.

• Anything on database migration tooling: Flyway, Liquibase, Alembic, or dbt snapshots. The schema half of §27.6 is solved and the solutions are mature. Adopt one rather than writing migrations by hand, and read its documentation on irreversible migrations specifically, which is where the interesting constraints are. Tier 1.

• Martin Fowler's writing on "expand and contract" / parallel change. Chapter 17's compatibility material and §27.6's forward-only migrations are the same idea, and Fowler's articulation is the one most people have read, which makes it useful shared vocabulary in a design discussion. Tier 2 — martinfowler.com; the articles are stable.

On testing with data

• The property-based testing literature — Hypothesis (Python) and QuickCheck. Not usually applied to data pipelines and worth considering: a property test generates inputs including the pathological ones you did not think of, which is §27.3's admitted gap in synthetic data. Start with an invariant — revenue is never negative, the grain holds, a dedup is idempotent — and let it generate. Tier 1 for Hypothesis's documentation.

• Great Expectations, and Chapter 23's assessment of it. Its place in CI is validating inputs before they land, which is the one thing dbt tests cannot do.

• Anything careful on data masking and synthetic generation. The tooling here is immature and vendor-heavy, and the honest position is that Kestrel's seed_kestrel.py — a few hundred lines of deliberate generation — outperforms most of it, because the value is in encoding the pathologies you know about, and only you know those. Tier 2 by nature; evaluate skeptically.

On the failures

• Anything on controls that fail open, from safety engineering. Case Study 2's "a check whose failure is indistinguishable from having nothing to check" is a fail-safe versus fail-open distinction, and the safety literature is precise about it in a way software writing rarely is. Nancy Leveson's Engineering a Safer World (MIT Press, 2011) is the serious treatment; the relevant idea is that a control's failure mode is a design choice and is usually made implicitly. Tier 1, and heavier going than anything else on this list.

• The literature on requirements phrasing, or failing that, the discipline of writing acceptance criteria positively. Case Study 2's "do not X; instead Y" finding is a specific, checkable version of a general problem, and the mitigation — state the positive clause first and test the negative path — is worth applying beyond CI. Tier 2 — scattered; the finding is more useful than any source.

Practice

• code/pr_report.py in this chapter. Computes blast radius, exposures, deploy shape, rebuild estimate, and margin impact from two dbt manifests, with no dbt and no warehouse. --demo shows an additive and a definitional change side by side, which is Case Study 1's distinction in one command.

• Break your CI on purpose. Exercise 27.14. Point state:modified+ at a missing manifest and watch it pass in eleven seconds. It takes two minutes and it is the most convincing argument in this chapter.

• Run pr_report.py against your last ten merged changes and count the definitional ones that were never rebuilt. Exercise 27.24. Expect at least one.

A note on what to be skeptical of

Any data CI/CD guide that does not mention the data half of a deploy. Most do not, because most are written about shipping models rather than changing what a table means, and the two are different problems with the same button.

And "just use a staging environment." §27.5's dilemma is real and has no clean answer; a guide that recommends staging without saying what data is in it has not engaged with the question. The follow-up to ask is: does your staging environment contain the row that caused your last incident? For most teams the answer is no, and it was not a close call.