Further Reading: The Data Engineering Lifecycle

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This chapter borrowed its central framework from one book, so that book dominates the list. The other entries are for the specific claims the chapter makes in passing.

The source of the framework

Joe Reis and Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022), Chapter 2. The lifecycle in its original form, with more room than this chapter gives it — particularly on the undercurrents, which get a fuller treatment there. Read it if you want the framework argued rather than asserted. Their Chapter 3, on architecture, extends directly into this book's Chapter 3. Tier 1.

On source systems and the Generate stage

PostgreSQL documentation, "Concurrency Control" (Chapter 13 of the manual). The section on transaction isolation and MVCC is what makes the watermark failure in §2.3 comprehensible rather than mysterious: a row's updated_at is a value the application wrote, and its visibility is governed by commit order, and those are two different clocks. Twenty minutes, and it is the foundation for Chapters 7, 13, and 14. Tier 1.
Martin Kleppmann, Designing Data-Intensive Applications, Chapter 7 ("Transactions"). The broader treatment of the same idea, vendor-neutral, and the best available explanation of why "read committed" is not the guarantee most people think it is. Tier 1.

On storage as substrate

The Apache Parquet format specification (parquet.apache.org/docs/file-format/). Short and worth skimming now rather than at Chapter 11, because the layout concepts in §2.4 — row groups, column chunks, pages, statistics — are what makes partition pruning and predicate pushdown possible at all. You do not need to understand the encoding details. Tier 2 — the docs site has been reorganized more than once.
AWS S3 documentation on request rates and performance. The primary source for the small-files arithmetic in §2.4. Read for the request-cost model and the per-prefix throughput characteristics rather than for the tuning advice, which changes. Tier 2 — cloud docs and their prices move constantly, and the figures in this book are frozen at the time of writing.

On transformation and grain

Ralph Kimball and Margy Ross, The Data Warehouse Toolkit, 3rd edition, Chapter 1. The concept of grain — declared before columns are chosen, never after — is the single most useful idea in this chapter's §2.5, and Kimball states it more forcefully than anyone since. "Declare the grain" is the second of his four steps in dimensional design, and the discipline of doing it second, before choosing dimensions or facts, is what prevents the join failures described here. Tier 1.
Maxime Beauchemin, "Functional Data Engineering — a modern paradigm for batch data processing" (2018). An essay arguing that batch pipelines should be built from pure, idempotent, deterministic functions over immutable partitions — reproducible by construction. It is the intellectual foundation for this book's insistence on idempotency and for the medallion pattern in Chapter 34. Read it before Chapter 20. Tier 2 — published on Medium and has moved.

On serving

Benn Stancil's writing on metrics layers and the semantic layer problem. The clearest available thinking on why the "define each metric exactly once" problem in §2.6 has resisted solution for two decades despite being obviously desirable, and why the technical answers keep failing for organizational reasons. Opinionated and worth disagreeing with. Tier 2 — blog content; find the current home.
Google Cloud, "Data lifecycle" and the equivalent architecture-center pages at AWS and Azure. All three cloud vendors publish a lifecycle model, and comparing them against Reis and Housley's is genuinely instructive: they agree substantially on the stages and diverge exactly where each vendor's products are strongest. That divergence is the most useful thing about reading them. Tier 2 — vendor content, rewritten often.

On the undercurrents

Google SRE Team, Site Reliability Engineering, free at sre.google/books. The DataOps undercurrent is SRE practice applied to data. The chapters on monitoring philosophy and on postmortem culture are the two to read now; Chapters 25 and 26 of this book build on both. Tier 1.
The GDPR text itself, Articles 12 and 17 (gdpr-info.eu or the official EUR-Lex text). Article 17 is the right to erasure and Article 12 sets the one-month response window referenced in §2.7. It is worth reading the actual articles once — they are shorter and clearer than most summaries of them, and the summaries frequently get the extension provisions wrong. This is not legal advice; confirm your own obligations with counsel. Tier 1.

If you only read one thing

Read Beauchemin's "Functional Data Engineering." It is about twenty minutes, it changes how you think about what a pipeline is, and every subsequent chapter in this book gets easier once the idea of an idempotent function over immutable partitions is in your head.