Affiliate disclosure

Book titles on this page link to Amazon. As an Amazon Associate, DataField.Dev earns from qualifying purchases — at no additional cost to you.

Further Reading: The Modern Data Stack

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This chapter has the shortest shelf life in the book, so this list leans toward writing about how to think about tools rather than writing about particular tools, which ages badly.

On the landscape and its incentives

  • Jordan Tigani, "Big Data Is Dead" (2023), MotherDuck blog. Argues from BigQuery usage data that most organizations have far less data than they believe and that single-node tools now handle the majority of real workloads. §5.7 is broadly in agreement. Read it with the disclosure in mind — the author's company sells single-node analytics — which is exactly the reading posture this chapter recommends for all vendor-adjacent writing, including the ones you agree with. Tier 2.

  • Benn Stancil's newsletter. The most consistently thoughtful writing about the data tooling industry as an industry — where the money is, what the incentives are, why categories appear and merge. Useful specifically as an antidote to reading tool documentation as though it were neutral. Tier 2 — newsletter; find the current home.

  • The awesome-data-engineering style lists on GitHub, and the various "data stack landscape" diagrams vendors publish annually. Genuinely useful for one purpose only: seeing the shape and size of each category at a glance. Do not read them as recommendations — a landscape diagram showing 300 logos is an argument that the category is confusing, which is itself information. Tier 2, and they go stale within a year.

On the tools this book uses

  • The dbt documentation (docs.getdbt.com). Among the best documentation in this field, and the "dbt Fundamentals" course is free and worth the few hours. Read it before Chapter 19 if you want a head start. Tier 1.

  • The Apache Airflow documentation, especially "Best Practices" and "Concepts." Airflow's docs have improved substantially and the Best Practices page in particular is unusually candid about what Airflow is bad at. Always check the version selector — the 2.x and 3.x documentation differ in ways that will cost you an afternoon if you read the wrong one. Tier 2 — versioned.

  • The DuckDB documentation (duckdb.org/docs). Short, well-written, and the "Guides" section is a genuinely good SQL reference in its own right. The blog is also worth following for the performance write-ups, which are unusually honest about where DuckDB is not the right tool. Tier 1.

  • The MinIO documentation on S3 API compatibility. Specifically the compatibility matrix, which lists exactly which S3 operations MinIO implements. Worth a glance so you know the boundary of the substitution before you hit it. Tier 2.

  • The Docker Compose specification (docs.docker.com/compose/compose-file/). The reference for the file in §5.6. The sections on depends_on conditions and healthcheck are the two that matter here and the two most people skip, which is why "works on the second run" is such a common experience. Tier 1.

On buy versus build

  • Joel Spolsky, "In Defense of Not-Invented-Here Syndrome" (2001). Twenty-plus years old and the core argument is undiminished: build what is core to your business, buy what is not, and the hard part is being honest about which is which. Data ingestion connectors are almost never core. Your dimensional model always is. Tier 2 — Joel on Software; the site is still up but has moved around.

  • Martin Fowler, "Utility vs Strategic Dichotomy" and the related writing on technical differentiation. The same argument in a more general frame, and useful for the conversation with a manager who wants everything built in-house or everything bought. Tier 2.

On operational burden, which is the chapter's real subject

  • Google SRE Team, Site Reliability Engineering (free at sre.google/books), the chapter on toil. The definition of toil — manual, repetitive, automatable, tactical, devoid of enduring value, and scaling linearly with service growth — is precisely the 372 hours in this chapter's first case study, and the SRE guidance to cap toil at 50% of a team's time is a useful benchmark even where it is unachievable. Tier 1.

  • Charity Majors' writing on operational ownership and on-call. Blunt, experienced, and the best available argument that the people who build a system should operate it — which is the principle under both the bus-factor discussion in Case Study 1 and the systems-per-engineer ceiling. Tier 2 — blog and conference talks.

On the "you might not need it" position

  • Dan McKinley, "Choose Boring Technology" (2015). The innovation-tokens essay. The argument is that an organization has a small budget for novelty and should spend it where it differentiates, running everything else on technology so well understood that it is uninteresting. It is the clearest statement of the position §5.5 and §5.7 argue from, and it is four pages. Tier 2 — personal site plus a well-known slide deck; both findable by title.

  • The PostgreSQL documentation on partitioning, parallel query, and BRIN indexes. Read as a companion to §5.7: these are the features that extend how far a single PostgreSQL instance takes you, and knowing they exist changes where you place the threshold for needing a warehouse. Tier 1.

If you only read one thing

Read "Choose Boring Technology." It is short, it is about a habit rather than a fact, and it gives you the vocabulary — innovation tokens — to have the conversation this chapter is really about: not which tool is best, but how many novel things a team can afford to be doing at once.

Then, if you have another hour, read the SRE toil chapter and apply its definition to whatever you currently operate. Most people are surprised by the total.