Further Reading: What Is Data Engineering?

Sources are tagged Tier 1 (we are confident the work exists and recommend it without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself — this field's documentation drifts fast enough that a link printed today may not resolve next year).

This is an orientation chapter, so most of what follows is orientation material. Do not try to read it all. Pick one or two entries that match what you want next; three of these are books, and reading three books before Chapter 2 is a way of not starting.

The three that shaped this book

• Joe Reis and Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022). The book that gave the field its organizing frame: data engineering as a lifecycle — generate, ingest, store, transform, serve — with undercurrents beneath every stage. Read it alongside this one; it is broader and more conceptual where this book is narrower and more hands-on, and Chapter 2 here is a deliberate restatement of their framework applied to one concrete system. If you buy one other data engineering book, buy this. Tier 1.

• Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017). The best explanation available of why distributed data systems behave the way they do — replication, partitioning, consistency models, consensus, stream processing. It is the reason Chapter 4 of this book exists, and where you should go when Chapter 4's compression to a single chapter leaves you wanting the real treatment. Chapters 5–9 are the load-bearing ones for a data engineer. Widely regarded as the most useful technical book of its decade, and the reputation is earned. Tier 1.

• Ralph Kimball and Margy Ross, The Data Warehouse Toolkit, 3rd edition (Wiley, 2013). The dimensional modeling canon: star schemas, conformed dimensions, fact table grain, slowly changing dimensions. Written for an era of hardware that no longer exists and almost entirely undiminished by that, because the modeling ideas were never really about the hardware. Chapters 6 and 20 of this book are Kimball with different syntax. Read Chapter 1 and Chapter 2 of it before this book's Chapter 6. Tier 1.

Where the modern architecture came from

• Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, "The Google File System" (2003), SOSP. And Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters" (2004), OSDI. The two papers that started the big data era. Read them for the reasoning rather than the systems: both are unusually clear about the assumptions — commodity hardware fails constantly, files are huge and append-mostly, moving computation beats moving data — and watching a design follow from stated assumptions is the transferable skill. Both are short and freely available. Tier 1.

• Fay Chang et al., "Bigtable: A Distributed Storage System for Structured Data" (2006), OSDI. The ancestor of most wide-column stores (HBase, Cassandra's data model, and their descendants). Relevant to Chapter 12. Tier 1.

• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013). A long essay, originally on LinkedIn's engineering blog, arguing that an append-only ordered log is the primitive underneath databases, replication, and stream processing alike. It is the clearest available explanation of why Kafka is shaped the way it is, and it reframes Chapters 14, 15, and 36 before you read them. Freely available; search the title. Tier 2 — the original hosting has moved more than once.

On the failure modes

• Google SRE Team, Site Reliability Engineering (O'Reilly, 2016), free online at sre.google. Not a data book, and the chapters on service level objectives, error budgets, alerting philosophy, and blameless postmortems transfer directly. Chapter 26 of this book is SRE practice adapted to a domain where the failures are silent. Start with the SLO chapter and the postmortem chapter; skip the ones about serving infrastructure. Tier 1.

• Richard Cook, "How Complex Systems Fail" (1998/2000). Eighteen numbered propositions, four pages, from a physician studying failure in medicine. Proposition 3 — catastrophe requires multiple failures; single-point failures are not enough — describes the duplicate-rows incident in Case Study 2 precisely: a missing DELETE, a scheduler change without review, no grain test, and monitoring aimed at pipelines rather than data. Read it in ten minutes; it will change how you write incident reviews. Freely available. Tier 1.

On the shape of the job

• DBT Labs, "What is analytics engineering?" and the accompanying discourse. Vendor material, and worth reading anyway as the clearest statement of the role boundary discussed in §1.4 — from the people who created the role's name and had every interest in defining it sharply. Read it aware of the interest. Tier 2 — vendor content moves and gets rewritten.

• Maxime Beauchemin, "The Rise of the Data Engineer" (2017) and "The Downfall of the Data Engineer" (2018). Two essays from the creator of Airflow and Superset, and the second is the better one. It describes the specific ways the role becomes miserable — being the bottleneck for every request, owning quality without owning the sources, the "data janitor" trap — and is the most honest thing written about the job's failure modes as a career rather than as a system. Read before you accept an offer. Chapter 40 covers this ground. Tier 2 — both are on Medium and have moved.

Reference and lookup

• PostgreSQL documentation, postgresql.org/docs. Genuinely excellent, unusually well-written technical documentation, and the reference you will open most across Part II. The chapters on indexes, EXPLAIN, and MVCC are the ones that repay reading straight through rather than looking up. Tier 1.

• The Apache Airflow, Kafka, Spark, and Iceberg documentation sets. Each is the primary source for its chapter (24, 15, 21, 10). Quality varies — Kafka's is the strongest, Airflow's has improved substantially, Spark's configuration reference is comprehensive and nearly unreadable as prose. Always check the version selector before trusting a page. Tier 2, unavoidably: all four version their docs and all four have broken links across major versions.

If you only read one thing

Read Kleppmann's Chapter 1 ("Reliable, Scalable, and Maintainable Applications"), about twenty pages. It defines reliability, scalability, and maintainability in a way that gives you vocabulary for arguments you are about to have, and it does it without requiring anything from the rest of the book.

If you have thirty minutes rather than two hours, read Cook's "How Complex Systems Fail" instead and then reread Case Study 2 with the eighteen propositions in hand.