Further Reading: Data Architecture Principles

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Architecture writing has a high ratio of opinion to evidence, which is not a criticism — most of it is judgment. The entries below are chosen for the quality of the reasoning rather than for agreeing with this chapter, and two of them disagree with it.

On architectural decisions and reversibility

• Michael Nygard, "Documenting Architecture Decisions" (2011). The essay that introduced the ADR and defined the format used in §3.7. Two pages. Its central argument — that the expensive thing to lose is not what was decided but why, and that the why evaporates within about eighteen months — is the entire justification for the practice. Read it before writing your first ADR. Freely available; search the title. Tier 2 — hosted on a personal blog that has moved.

• Martin Fowler, "Who Needs an Architect?" (2003), IEEE Software. The source of the "architecture is the decisions that are hard to change" framing, which Fowler attributes to Ralph Johnson. Short, and the more interesting half is the argument that a good architect's job is to reduce the number of decisions that are hard to change — which is exactly the "push decisions down the ranking" move in §3.1. Tier 1.

• Adrian Cockcroft's writing and talks on evolutionary architecture at Netflix. Useful as a counterweight to this chapter: Cockcroft's position is closer to "make everything replaceable and accept the operational cost," which is a defensible read of the same evidence at a very different organizational scale. Read it noticing the scale, which is the discipline §3.1 asks for. Tier 2 — talks and blog posts, scattered.

On batch, streaming, and the space between

• Tyler Akidau, Slava Chernyak, and Reuven Lax, Streaming Systems (O'Reilly, 2018). The definitive treatment of event time, watermarks, windowing, and the "what / where / when / how" framework for stream processing. It is more rigorous than this book's Chapter 29 and it is where to go when you need the real thing. Chapters 1 and 2 are the highest-value part and are readable independently. Tier 1.

• Tyler Akidau, "Streaming 101" and "Streaming 102" (2015), on the O'Reilly Radar blog. The essays that became the book. If two hours is what you have, read these instead. "Streaming 102" is the one that will change how you think about windows. Tier 2 — the O'Reilly blog has been reorganized.

• Jay Kreps, "Questioning the Lambda Architecture" (2014). The essay that named and then argued against the pattern of running parallel batch and streaming paths, proposing what became known as Kappa. It is a good example of an architecture argument conducted well: it states the opposing position fairly before dismantling it. Chapter 29 §29.2 revisits this debate with a decade of hindsight. Tier 2.

On warehouse, lake, and lakehouse

• Michael Armbrust et al., "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics" (2021), CIDR. The paper that named the lakehouse. It is written by Databricks people about a category Databricks sells into, and it is still the clearest statement of the technical argument — read it for the problem framing and discount the product conclusions accordingly. Tier 1.

• Michael Armbrust et al., "Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores" (2020), VLDB. How the transaction log actually works: optimistic concurrency, atomic commits over an eventually consistent object store, and why the naive approaches fail. This is the paper that makes the ⚠️ callout in §3.4 concrete. Tier 1.

• The Apache Iceberg specification (iceberg.apache.org/spec/). Worth reading alongside the Delta paper because the two solve the same problem differently — Iceberg's manifest-based metadata versus Delta's transaction log — and the comparison teaches you what is essential to the problem and what is one design's choice. Tier 2 — the spec is versioned and moves.

• Bill Inmon and Ralph Kimball's original disagreement, best encountered through Kimball's The Data Warehouse Toolkit Chapter 1 and Inmon's Building the Data Warehouse. The normalized-enterprise-warehouse versus dimensional-bus-architecture argument is thirty years old, was never fully resolved, and both positions are visible in modern platforms. Chapter 6 takes Kimball's side and explains why. Tier 1.

On sizing and capacity

• John D. C. Little's original 1961 paper, "A Proof for the Queuing Formula $L = \lambda W$", Operations Research. Short, and the proof is more general than most people realize — it requires almost no assumptions about arrival or service distributions, which is exactly why it is safe to apply to a Kafka consumer. Read it once to understand why the formula is trustworthy. Tier 1.

• Neil Gunther, Guerrilla Capacity Planning (Springer, 2007). Capacity planning for people who do not have a queueing theory background and do have a deadline. The universal scalability law is the most useful idea in it, and it explains why adding nodes stops helping — and eventually starts hurting — which §3.5 gestures at and does not develop. Tier 2 — check for the current edition.

Two that disagree with this chapter

• Jordan Tigani, "Big Data Is Dead" (2023), MotherDuck blog. Argues, with data from BigQuery usage, that the overwhelming majority of organizations have far less data than they think and that single-node tools now handle most real workloads. This book broadly agrees, which is why Chapter 22 exists — read it also for the honest disclosure that the author's company sells single-node analytics. Tier 2.

• Any vendor's "why the data warehouse is dead" or "why the data lake failed" content. Read two of these from competing vendors back to back. The exercise is genuinely instructive: they make opposite arguments from the same evidence, and seeing that clearly is a useful inoculation against architecture writing generally, including this chapter. Tier 2, and the tier is doing a lot of work here.

If you only read one thing

Read Nygard on ADRs. Two pages, and it changes a habit rather than adding knowledge. The architecture decisions you make this year will be re-litigated by someone in 2029 who has no idea what you rejected, and the only defense is two hours of writing at the time.