Exercises: The Modern Data Stack

Half of these are judgment and half are Docker. Part D is the first exercise in this book that produces running infrastructure — do it even if you skip everything else.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

5.1 † Name the eight tool categories and state in one clause the problem each solves.

5.2 The chapter gives three reasons the term "modern data stack" is unhelpful. Name them.

5.3 † State the systems-per-engineer heuristic from §5.1. Apply it to a team of six operating fourteen systems. What does the heuristic say, and what is its most obvious weakness?

5.4 For each category, say whether Kestrel buys, builds, or does without: ingestion from its own PostgreSQL, ingestion from a SaaS CRM, the warehouse, transformation, catalog, BI.

5.5 † Which two categories does §5.3 identify as the most expensive to lack, and why each?

5.6 Name three ways DuckDB differs from a cloud warehouse in a way that matters for this book.

5.7 † In the Kestrel docker-compose.yml, why is wal_level=logical set in Chapter 5 when CDC does not arrive until Chapter 14?

5.8 What is the single line of Python that differs between talking to MinIO and talking to S3?

Part B — Standard ⭐⭐

5.9 Redo the §5.3 managed-ingestion analysis for a different company: 400,000 orders a year, 2.1 items per order, 60,000 customers, no clickstream, and eleven SaaS sources (CRM, support desk, ads, email, billing, and so on). Estimate monthly changed rows. Then state which of buy / build / do without you would choose for each of the three source groups, and why the answer differs from Kestrel's.

5.10 † The minio-init service waits on condition: service_healthy rather than the default. Explain the specific failure that the default produces, why it usually works on the second run, and why "usually works on the second run" is one of the more dangerous properties a system can have.

5.11 Take three tools you have not used — pick from Dagster, SQLMesh, Soda, OpenMetadata, Hightouch, Temporal, Trino. For each, answer §5.8's criteria 2 (exit cost) and 3 (price scaling) from public information alone. Note which you could not answer, and what that tells you.

5.12 † Write the .env.example file for Kestrel's compose stack. It must document every variable the compose file requires, contain no real values, and be safe to commit. Then explain why .env.example is committed and .env is not, to someone who thinks that is redundant.

5.13 §5.7 describes an architecture with no data platform at all: PostgreSQL, scheduled SQL, and a BI tool. Draw the version of that architecture for Kestrel and identify the exact point at which it breaks. Name the metric you would monitor to see that point coming.

5.14 † The 🏭 callout describes nine months spent to make a query two seconds faster. Write the one-page memo that the engineer should have written before the project started. It must contain the question from the callout's last paragraph and its answer.

Part C — Deeper ⭐⭐⭐

5.15 The systems-per-engineer heuristic (§5.1) says a four-person team can operate about eight systems. Construct the strongest argument against it. Consider: managed services that reduce operational burden, systems that genuinely never break, and the difference between "operating" and "depending on." Then propose a better heuristic and test it against Kestrel.

5.16 † Kestrel does without a catalog and accepts that a Markdown file is worse. Quantify the gap: name five specific things a catalog product would do that the Markdown file does not, and for each, estimate how often it matters at Kestrel's size. At what team size does the balance flip?

5.17 §5.4 says the DuckDB substitution "leaks" on concurrency, storage/compute separation, and cost. Design an exercise that would let a reader experience one of those three gaps rather than just read about it. It must be runnable locally and must fail in an instructive way.

5.18 † Take the seven evaluation criteria in §5.8 and apply all seven to dbt, honestly, including the ones where the answer is uncomfortable. Then do the same for Airflow. Which of the two would you find harder to leave, and does that change your view of adopting it?

Part D — The Kestrel Platform ⭐⭐⭐

5.19 — Increment 5: the stack comes up.

(a) Create platform/infra/docker-compose.yml from §5.6. Type it rather than copying it — you will be editing this file in eight more chapters and you want to know what is in it.

(b) Create platform/infra/.env.example documenting every required variable, and platform/infra/.env with real values. Confirm .env is gitignored, and confirm it by running git status and seeing that it does not appear.

(c) docker compose up -d, then verify each service independently using the three checks in the 🧱 callout. Do not accept "the containers are running" as verification — a running container and a working service are different claims.

(d) Write platform/infra/verify_stack.py from the callout, and run it. It must print the four bucket names. Extend it: have it also connect to PostgreSQL, confirm wal_level is logical, and exit non-zero if anything is wrong. This script is your smoke test for the rest of the book, and you will run it every time something behaves strangely.

(e) Commit all of it.

5.20 † Add platform/docs/tool-decisions.md. For each of the eight categories, record: what Kestrel uses, whether that is buy / build / do without, one sentence of reasoning, and — the part that matters — what you are giving up. Where the answer is "do without," write down the observable condition that would change your mind.

This is a shorter, lighter artifact than an ADR, and it is deliberately per-category rather than per-decision. Chapter 38 asks you to reread it.

Reflection

A. §5.1 argues that most tooling advice is written by people at a different scale than yours. Think of technical advice you have followed that turned out to be scale-specific. What signal, if any, would have told you at the time?

B. The 🧪 Try It in §5.8 asks you to evaluate a tool you already use against seven criteria. Which question was hardest to answer, and what does it say about how the tool was chosen?