Chapter 5: The Modern Data Stack

"The best tool is the one your team can operate at three in the morning."

Overview

This is the most perishable chapter in the book, and it is worth saying so at the top.

Every product named here may be gone, acquired, renamed, or superseded within a few years. Two of the categories described did not exist a decade ago and one of them may not exist a decade from now. If you are reading this some years after publication, treat the specific names as illustrative and the categories as the content — because the categories are stable in a way that the products are not.

So the chapter is organized around a different question than "what should I use." It is organized around what problem does each category of tool exist to solve, what does it cost to operate, and what happens if you simply do not have one. That last question is the one nobody asks, and it is frequently the most valuable, because the answer for a small team is often "nothing much."

There is a bias to correct for here, and it runs through almost all writing about tooling. The people who write about data tools are disproportionately people who work at data tool companies, who speak at conferences sponsored by data tool companies, or who work at companies large enough to have a team per category. Their advice is not dishonest and it is calibrated to a situation that is not yours. A four-person team does not need — and cannot operate — the stack described in most architecture diagrams.

The chapter ends with the Kestrel platform's first running infrastructure: a docker-compose.yml with PostgreSQL and object storage, which is the point at which this book stops being about ideas.

In this chapter, you will learn to:

• Name the eight categories of tool in a data platform and the problem each solves.
• Explain why "the modern data stack" is a marketing term and what the framing obscures.
• Apply a buy / build / do without decision using operational cost rather than licence cost.
• Identify what a small team can safely leave out, and price what leaving it out costs.
• Stand up the Kestrel local stack and verify it.
• Recognize the point below which none of this is needed, and make the case for PostgreSQL and a cron job without embarrassment.
• Evaluate a tool with criteria that survive a vendor demonstration.

Who needs this chapter: everyone, once. It is the chapter to re-skim rather than reread, and the one whose specific claims you should verify against the current landscape.

5.1 The Term Is a Problem

"The modern data stack" names a real thing: a cluster of tools that emerged around 2015–2020, sharing an architectural premise — cloud-native, separated storage and compute, SQL-centric transformation, and ELT rather than ETL.

It is also a marketing term, coined and promoted by vendors who sell components of it, and it does three unhelpful things.

It implies a package. The phrase suggests these tools belong together, that adopting one implies adopting the others, and that a "modern" platform has one of each. None of that follows. You can use dbt without Fivetran, Airflow without Snowflake, and Snowflake without any of it.

It implies "modern" versus "legacy." This framing is doing rhetorical work. A well-run PostgreSQL database with scheduled SQL transformations is not legacy — it is appropriate, for a large number of companies, and calling it legacy is how organizations end up with eleven services and four engineers.

It obscures the operational cost. Every architecture diagram shows boxes and arrows. None shows who gets paged when each box breaks. The diagram is the easy part; the pager rotation is the constraint.

📐 Design Decision — Count the boxes, then divide by the team

A useful heuristic when evaluating any architecture, including the ones in this book: count the distinct systems that can page someone, and divide by the number of engineers.

A four-person team running eleven systems has 2.75 systems per person. Each of those systems needs someone who understands its failure modes, its upgrade path, its configuration, and its recovery procedure — and needs a second person who can cover when the first is on vacation. At 2.75 systems per person, you have no depth anywhere.

Roughly two systems per engineer, with genuine coverage on each, is a sustainable ceiling. A four-person team can operate about eight systems well. Kestrel's platform, counted honestly, is seven: PostgreSQL, object storage, Kafka, the warehouse, Airflow, the transformation framework, and the monitoring stack.

What this heuristic costs you: it will tell you not to adopt tools that would genuinely help. A catalog would make Kestrel better and the team does not have room for it, so they use a Markdown file and accept that it is worse. That is a real loss, honestly taken, rather than a pretense that the Markdown file is just as good.

What is genuinely different about the current generation

Stripping the marketing, three things did actually change and they are worth naming because they are the reason the tools look the way they do:

Storage and compute separated. Covered in Chapter 3 §3.3. This is the load-bearing change; almost everything else follows from it.

Transformation moved into version control. dbt's contribution, and the reason it spread faster than its technical merits alone would explain.

Managed services became the default rather than the exception. You can now rent every component, which changes the build-versus-buy calculation and introduces a new failure mode: a platform assembled entirely from rented parts whose costs compound and whose integration is your problem.

5.2 The Eight Categories

Every data platform has these functions, whether or not it has a tool for each. Some are performed by a product, some by a script, and some not at all.

   ┌─────────────┐   ┌─────────────┐   ┌──────────────┐   ┌─────────────┐
   │ 1 INGESTION │──▶│  2 STORAGE  │──▶│3 TRANSFORM   │──▶│  7 BI /     │
   │             │   │             │   │              │   │  8 REVERSE  │
   └─────────────┘   └─────────────┘   └──────────────┘   └─────────────┘
          ▲                 ▲                  ▲                 ▲
          └─────────────────┴──────────────────┴─────────────────┘
                              │
              ┌───────────────┼───────────────┬────────────────┐
              │               │               │                │
        ┌───────────┐  ┌────────────┐  ┌────────────┐  ┌───────────────┐
        │4 ORCHEST- │  │ 5 QUALITY  │  │6 CATALOG / │  │  (monitoring, │
        │  RATION   │  │            │  │ GOVERNANCE │  │   Ch. 25)     │
        └───────────┘  └────────────┘  └────────────┘  └───────────────┘

In words: four categories sit in the data flow (ingestion, storage, transformation, and the serving categories), and four sit beside it, applying across all stages.

1. Ingestion

The problem: getting data from source systems into your platform, reliably, incrementally, and without breaking when a schema changes.

Managed options: Fivetran, Airbyte, Stitch, Meltano. They maintain connectors to hundreds of sources so you do not have to.

The trade-off is unusually stark. Connectors are genuinely tedious to write and maintain — pagination quirks, auth refresh, rate limits, schema drift — and a managed connector to Salesforce saves real weeks. But managed ingestion is typically priced per row or per "monthly active row," which means your bill scales with your data volume rather than with the value you get, and at high volume this becomes the largest line on a data platform's invoice.

The practical pattern most teams converge on: buy connectors for third-party SaaS sources, build for your own systems. A Salesforce connector is worth paying for. An extract from your own PostgreSQL is a hundred lines of code you fully understand.

Kestrel builds its own for all three sources — Chapters 13 through 16 — because two of the three are its own systems and the third is one API.

2. Storage

Covered in Part II. Categories: relational (Chapter 7), warehouse (Chapter 8), object storage / lake (Chapter 9), lakehouse table formats (Chapter 10), and specialized stores (Chapter 12).

This is the category where the buy decision is easiest: almost nobody should operate their own warehouse. The managed options are good, the operational burden of self-hosting is severe, and the differentiation is not in your favor.

3. Transformation

The problem: turning raw data into modeled, tested, documented tables.

The dominant answer is dbt, which compiles templated SQL and executes it in dependency order, with tests, documentation, and lineage generated from the same source. Alternatives: SQLMesh, Dataform, hand-rolled SQL with a runner, or Spark for the cases SQL cannot express.

The reason to use something rather than nothing is not the SQL compilation. It is that a transformation framework forces your business logic into version-controlled files with a dependency graph, which is what makes it reviewable, testable, and comprehensible to the next person. Chapter 19.

4. Orchestration

The problem: running things in the right order at the right time with the right failure behavior.

Airflow is the default. Alternatives worth knowing: Dagster (asset-oriented rather than task-oriented, and the better model in most respects), Prefect, Temporal for durable execution, and — genuinely — cron for a small enough platform.

What an orchestrator buys you over cron: dependencies expressed rather than implied by sleeps; retries with backoff; backfills as a first-class operation; visibility into what ran and what failed; and a place for the alerting to live. That last one matters more than people expect.

Chapter 24.

5. Data quality

The problem: knowing whether the data is right, not just whether the job ran.

Options: dbt's built-in tests (free, adequate for most cases, and where you should start), Great Expectations (richer, heavier), Soda, Monte Carlo and similar observability products (which infer expectations rather than requiring you to state them).

Start with dbt tests. They cover uniqueness, not-null, referential integrity, and accepted values, which is the large majority of what catches real problems, and they cost nine lines of YAML. Add more when you can name what the added tool would catch that these do not. Chapter 23.

6. Catalog and governance

The problem: knowing what data exists, what it means, where it came from, and who can see it.

Options: DataHub, OpenMetadata, Amundsen, Unity Catalog, and the cloud vendors' offerings.

This is the category most often skipped and most often regretted. It is skipped because the pain is diffuse — nobody is blocked today by not having a catalog — and regretted because the pain compounds: five teams building the same table, a deletion request that cannot be fulfilled, and a new engineer taking three months to become productive.

The cheap version genuinely works for a while. dbt generates documentation and a lineage graph from your model definitions at zero additional cost, and a Markdown file listing every table with its owner and grain covers most of what a small team needs. Chapter 30.

7. BI and analytics

The problem: letting people see and explore the data.

Options: Looker, Tableau, Power BI, Metabase, Superset, Hex, Evidence.

This is the category you should care least about and your stakeholders will care most about, which is a recurring source of misallocated effort. A good model behind a mediocre BI tool is far better than the reverse, and the model is where your effort belongs.

8. Reverse ETL

The problem: getting warehouse data back into operational systems — a churn score into the support tool, lifetime value into the CRM.

Options: Census, Hightouch, or a Python script.

The newest category and the one that inverts your risk profile. A wrong number on a dashboard is embarrassing; a wrong number written into a customer-facing system is an incident. Test accordingly.

5.3 Buy, Build, or Do Without

Three options, and the third is systematically underweighted.

The decision, honestly framed

The usual buy-versus-build analysis compares licence cost against engineering time. That framing misses the two costs that actually dominate.

The cost of building is not the build; it is the maintenance. A connector you write in three days needs updating when the API changes, when auth rotates, when the schema drifts, and when the person who wrote it leaves. Over three years that is far more than three days.

The cost of buying is not the licence; it is the integration and the exit. Every purchased tool needs configuring, monitoring, upgrading, and securing. It has an outage model you do not control. And the exit cost — what it takes to leave — is invisible at purchase and decisive later.

💸 Cost Check — What a managed ingestion tool would cost Kestrel

Managed ingestion is typically priced per monthly active row. Rates vary substantially and change often, so treat the following as an illustration of the shape rather than a quote — and check current pricing before repeating any of it.

Kestrel's monthly changed-row volume:

Source	Rows changed per month
orders + order_items (2.4M orders/yr × 2.7 lines, plus status updates)	~1.2M
customers, products, inventory, other dimensions	~0.9M
Clickstream — 420M/month	not viable at per-row pricing

The transactional sources at roughly 2.1M monthly active rows sit in the range where managed ingestion is plausibly worth it: the engineering time saved is real, and the volume is moderate.

The clickstream is not. At 14M events a day, per-row pricing is out of the question by orders of magnitude, which is why high-volume event data is essentially never handled by row-priced ingestion tools. It goes through Kafka or a direct producer.

The generalizable lesson: per-row pricing has a volume ceiling above which it is simply the wrong product, and platforms typically end up with both — a managed tool for SaaS sources and a hand-built path for high-volume events. Budget for both rather than expecting one to cover everything.

The buy/build table, category by category

For a four-person team at Kestrel's scale:

The "do without" column

For each category, what does not having it actually cost?

Read the right-hand column and notice the ranking. Quality is the most expensive to lack, and transformation framework is the one whose absence compounds fastest. Those are the two to have before anything else. A catalog can wait. A reverse ETL tool can wait indefinitely.

5.4 What This Book Uses, and Why

Every tool in this book runs locally, free, in Docker. Where the local choice differs from what you would run in production, this section says so — because pretending the local stand-in is the real thing is how people are surprised in their first job.

🧭 Version Note — DuckDB is the biggest substitution in this book

DuckDB is an excellent analytical database and it is genuinely used in production for single-machine analytics. It is not a substitute for a cloud warehouse in three specific ways you need to know before an interview:

1. Concurrency. DuckDB is single-process with one writer. A cloud warehouse serves hundreds of concurrent queries across independently scaled compute clusters.
2. Storage and compute are not separated. The defining economic property of the modern warehouse (Chapter 3 §3.3) is absent — which means the cost-model reasoning in Chapters 8 and 33 has to be taught with reference to Snowflake and BigQuery rather than demonstrated locally.
3. No cost meter. You cannot learn cost discipline from a system that is free, so this book teaches it through arithmetic on the frozen basis instead.

What DuckDB is superb at, and why it is the right teaching choice: SQL semantics, dimensional modeling, dbt, incremental strategies, window functions, and reading query plans — all of which transfer completely. Chapter 8 is explicit about what changes on a real warehouse.

5.5 What You Can Safely Leave Out

A short, opinionated list of things a small team is frequently told it needs and does not.

A streaming platform, unless something automated consumes the stream. Chapter 3 §3.2 and its case study. Kestrel has Kafka because it has genuine event volume and a CDC pipeline; a company with neither should not.

A feature store, until you have models in production sharing features. A feature store solves training/serving skew and feature reuse across teams. With one model and one team, a well-modeled warehouse table is a feature store. Chapter 32 is explicit about the threshold.

A data mesh, until the central team is genuinely a bottleneck and you have real domain boundaries. Chapter 35 takes this seriously and is direct: most data mesh implementations are org charts wearing an architecture costume.

A dedicated observability product, before you have basic freshness and volume checks. These products are good. They are also frequently bought to substitute for practice rather than to extend it, and a team with no freshness checks and a Monte Carlo licence has bought a smoke detector for a house with no fire exits.

Kubernetes, unless you already run it for something else. Running data workloads on Kubernetes is reasonable if the platform exists and someone operates it. Standing it up for data workloads, at a four-person team, is a project that will consume more than it returns.

A semantic layer, before you have one place metrics are defined. The problem a semantic layer solves is real (Chapter 2's Case Study 2). A dbt model with a documented definition solves most of it for free.

🎓 Interview Angle — "What tools have you used?"

Asked in almost every screen, and the answer that stands out inverts the question.

A tool list is what everyone gives. What distinguishes a strong candidate is naming a tool they chose not to use, and why:

"We used Airflow, dbt, and Snowflake. Worth mentioning what we didn't use — we deliberately didn't adopt a catalog product, because with four engineers we couldn't operate it well, and dbt docs plus an ownership table covered maybe 70% of the value for free. We knew that was worse, and we wrote down what we were giving up: mostly discoverability across teams, which mattered less at our size. If we'd grown past about fifteen people I'd have revisited it."

That answer shows you understand operational cost, that you make decisions rather than accumulate tools, and that you can state a trade-off honestly. It is a substantially better signal than knowing one more product name.

5.6 The Kestrel Local Stack

Here is where the book stops being about ideas.

The Kestrel platform runs entirely in Docker. This section stands up the first two services; subsequent chapters add the rest, and Appendix A has the complete file plus troubleshooting.

Two services now: PostgreSQL as the source system, and MinIO as object storage. Nothing else yet, on purpose — every service you add is a thing that can fail while you are trying to learn something unrelated.

# platform/infra/docker-compose.yml  — increment 1: source + storage
services:

  postgres:
    image: postgres:16.4
    container_name: kestrel-postgres
    environment:
      POSTGRES_DB: kestrel_app
      POSTGRES_USER: kestrel
      # No default. Compose reads this from platform/infra/.env, which is
      # gitignored. A literal here is the single most common way a credential
      # reaches a public repository. See Chapter 28 section 28.4.
      POSTGRES_PASSWORD: ${KESTREL_DB_PASSWORD:?set KESTREL_DB_PASSWORD in .env}
    command:
      # Logical replication, needed for CDC in Chapter 14. Setting it now
      # avoids a restart later, and a restart of a database with data in it is
      # a thing you want to have practiced before you need it.
      - "postgres"
      - "-c"
      - "wal_level=logical"
      - "-c"
      - "max_replication_slots=4"
      - "-c"
      - "max_wal_senders=4"
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kestrel -d kestrel_app"]
      interval: 5s
      timeout: 5s
      retries: 10

  minio:
    image: minio/minio:RELEASE.2024-11-07T00-52-20Z
    container_name: kestrel-minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:?set MINIO_ROOT_USER in .env}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?set MINIO_ROOT_PASSWORD in .env}
    ports:
      - "9000:9000"      # S3 API
      - "9001:9001"      # web console
    volumes:
      - minio-data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 5s
      retries: 10

  # Creates the bucket layout once, then exits. A one-shot init container is
  # the right shape for setup work: it is idempotent, it is visible in
  # `docker compose ps`, and it fails loudly rather than leaving you with a
  # running stack and no bucket.
  minio-init:
    image: minio/mc:RELEASE.2024-11-05T11-29-45Z
    depends_on:
      minio:
        condition: service_healthy
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000 $$MINIO_ROOT_USER $$MINIO_ROOT_PASSWORD &&
      mc mb --ignore-existing local/kestrel-bronze &&
      mc mb --ignore-existing local/kestrel-silver &&
      mc mb --ignore-existing local/kestrel-gold &&
      mc mb --ignore-existing local/kestrel-scratch &&
      echo 'buckets ready'
      "

volumes:
  postgres-data:
  minio-data:

Four things in that file are worth reading rather than skimming.

No password literals. ${KESTREL_DB_PASSWORD:?...} fails the startup with a clear message if the variable is missing, rather than silently defaulting. The values live in platform/infra/.env, which is gitignored, and .env.example documents what is needed without containing anything.

wal_level=logical is set now, though CDC does not arrive until Chapter 14, because changing it requires a database restart and it is better to have the setting in place before there is data you care about.

Health checks on both services, and minio-init waits for service_healthy rather than just service_started. The difference is the source of a very common "it works on the second run" problem: depends_on without a condition only waits for the container to start, not for the service inside it to accept connections.

Four buckets, not one. Bronze, silver, gold, and scratch. The layer separation from Chapter 34 begins at the storage boundary, and separate buckets make access control (Chapter 30) a bucket policy rather than a prefix policy.

🧱 Kestrel Platform — Increment 5: the stack comes up

bash cd platform/infra cp .env.example .env # then edit it: set real values docker compose up -d docker compose ps # all services healthy, minio-init exited 0

Verify each service independently rather than assuming:

```bash

PostgreSQL responds and has logical replication enabled

docker exec kestrel-postgres psql -U kestrel -d kestrel_app \ -c "SHOW wal_level;"

expect: logical

MinIO's S3 API answers, and the buckets exist

docker exec kestrel-minio mc ls local/

expect: kestrel-bronze/ kestrel-silver/ kestrel-gold/ kestrel-scratch/

```

Then the one that matters, because it is the interface everything else uses — talk to MinIO with boto3, exactly as you would talk to S3:

```python

platform/infra/verify_stack.py

import os import boto3

s3 = boto3.client( "s3", endpoint_url="http://localhost:9000", # the ONLY line that differs from S3 aws_access_key_id=os.environ["MINIO_ROOT_USER"], aws_secret_access_key=os.environ["MINIO_ROOT_PASSWORD"], region_name="us-east-1", ) buckets = [b["Name"] for b in s3.list_buckets()["Buckets"]] print(sorted(buckets))

['kestrel-bronze', 'kestrel-gold', 'kestrel-scratch', 'kestrel-silver']

```

That one differing line — endpoint_url — is the whole substitution. Everything you write against MinIO in this book runs against S3 by removing it and supplying real credentials. Commit the compose file and the verification script.

⚠️ Failure Mode — The three things that go wrong here

In order of frequency, from teaching this stack:

Port already in use. Something else is on 5432 — often a locally installed PostgreSQL. Symptom: bind: address already in use. Fix: change the host-side port mapping to 5433:5432 and connect to 5433, or stop the other service. Do not change the container side.

Docker memory limit. The full stack in later chapters needs more than Docker Desktop's default. Symptom: containers exit with code 137, which is an out-of-memory kill and looks like a crash. Fix: raise Docker's memory allocation to at least 8 GB. Appendix A §A.6.

Stale volumes after a config change. You change POSTGRES_DB and the change does not take effect, because the initialization script only runs on an empty data directory. Symptom: the old database name persists. Fix: docker compose down -v, which deletes the volumes — fine now, not fine after you have seeded data. Know which command destroys state before you need to run it in a hurry.

5.7 When You Need None of This

The most useful section in the chapter for a certain kind of reader.

If your entire analytical dataset fits comfortably in PostgreSQL — say, under a few hundred gigabytes — and your team is under about ten people, you may not need a data platform at all.

What works, and works well:

  application PostgreSQL
        │
        │  nightly pg_dump / logical replication / a scheduled INSERT ... SELECT
        ▼
  analytics PostgreSQL (a separate instance, or a separate schema)
        │
        │  scheduled SQL: cron, or pg_cron, or a 40-line Python runner
        ▼
  Metabase / Superset / a BI tool connected directly

This architecture:

• Costs one database instance and a BI tool.
• Is operable by one person who knows PostgreSQL.
• Has ACID transactions, real constraints, and a mature query planner.
• Handles tens of millions of rows without difficulty, and hundreds of millions with attention to indexing and partitioning.
• Can be tested, version-controlled, and code-reviewed exactly like any other software — and you can run dbt against it, getting most of Chapter 19's benefits with none of the platform.

The failure modes are real and worth naming: analytical queries competing with transactional ones (fix: a separate instance or a read replica); no separation of storage and compute, so you scale by buying a bigger machine; and difficulty with semi-structured or very high-volume event data, which PostgreSQL handles but not cheaply.

The threshold to move: when analytical query load starts affecting the application, when a single machine can no longer hold the data at acceptable cost, or when you need to combine sources that PostgreSQL cannot reasonably ingest. Not before. And notice that all three are observable conditions — exactly the shape a reversal condition should take (Chapter 3 §3.7).

🏭 From the Pipeline — The platform that was a rounding error

A company with about forty engineers and eighteen million rows of transactional data spent nine months building a data platform: object storage, Spark, an orchestrator, a warehouse, and a streaming pipeline. Two engineers full-time.

Their largest table was 18 million rows. Their most expensive query, on the new warehouse, ran in 4 seconds. On a PostgreSQL read replica with two indexes added, the same query ran in 6 seconds.

Nine months and two engineers, for two seconds.

The decision was not stupid, and this is the part worth understanding. It was made because a senior engineer had come from a company where the platform was necessary, and reproduced what they knew — without re-deriving whether the conditions that made it necessary were present. That is the most common mechanism by which over-engineering happens, and it is not a knowledge failure. It is a failure to check whether the premise transferred.

The check is one question, asked before the project: what is our largest table, and how long does our worst query take on the system we already have? Ten minutes.

5.8 Evaluating a Tool

You will evaluate tools for the rest of your career, usually with vendor material as your primary input. Seven criteria that survive a demonstration.

1. What happens when it breaks at 3am? Is there a runbook? Can you diagnose it from logs? Is there a support contract, and what is its actual response time? A tool nobody on your team can debug is a tool you cannot operate, regardless of what it does when it works.

2. What is the exit cost? If you stop using it in two years, what do you have to rebuild? A tool that stores your data in its own format has a high exit cost; one that reads and writes open formats has a low one. Ask this at purchase, when nobody wants to think about it, because it is invisible then and decisive later.

3. How does the price scale? Per row, per seat, per compute-hour, per connector, flat? Model it at 10× your current volume. Per-row pricing that is comfortable today can be untenable after a good year, and the demonstration will use today's volume.

4. Who else uses it at your scale? Not "who uses it" — Fortune 500 logos on a website tell you nothing about whether it works for four engineers. Find someone at roughly your size and ask them what broke.

5. What does it assume about your workflow? Every tool encodes opinions. dbt assumes SQL transformation in git. Airflow assumes Python DAGs. Fighting a tool's assumptions is a permanent tax, and the demonstration will show you the happy path where the assumptions fit.

6. Can you try it for real? Not a demo — a proof of concept with your data, your volumes, and your ugliest edge case. Every tool works on clean data. Bring the messy table.

7. What is the maintenance burden per year? Upgrades, migrations, breaking changes, configuration drift. Check the release history: how often are there breaking changes, and how painful were the last two major upgrades? This information is public and almost nobody looks.

🧪 Try It — Evaluate a tool you already use

Pick a tool you or your team currently uses. Answer all seven questions in writing, honestly, in twenty minutes.

Then answer the eighth: would you choose it again? If yes, you have just written the justification you will need next time someone proposes replacing it. If no, you have found a project — and, more usefully, you have found out which of the seven criteria it fails, which is what tells you whether the replacement would actually be better.

Most people, doing this for the first time, discover they cannot answer question 2 or question 7 about a tool they depend on daily.

Leaving a tool

Adoption gets a decision and a document. Departure gets neither, and the asymmetry is why platforms accumulate systems nobody would choose today.

The exit cost of a tool is not proportional to how much you use it. It is proportional to how much of your logic lives inside it, and those are different quantities.

tool                        heavily used?   exit cost   because
───────────────────────────────────────────────────────────────────────────
managed Postgres            yes             LOW         it speaks SQL and
                                                        pg_dump exists
object storage              yes             LOW         the files are the
                                                        interchange format
dbt                         yes             MEDIUM      models are SQL; ref(),
                                                        macros and tests are not
Airflow                     yes             HIGH        business logic ends up
                                                        INSIDE operators
a GUI workflow tool         maybe           VERY HIGH   the logic is not text
                                                        and cannot be diffed
a BI tool with a
  semantic layer            yes             VERY HIGH   metric definitions live
                                                        there, not in the DAG

The two very-high rows share a property: the logic is not in a file you own. That is the single best predictor of exit cost and it is checkable at adoption — can I read this system's behaviour in version control?

The four questions to ask before adopting

1. Where does the logic live? In files in my repository, or in the tool's database? A tool whose configuration is in its own database is a tool whose behaviour cannot be reviewed, diffed, or restored from a git history.

2. What is the interchange format on the way out? Not "does it have an export button" — has anyone run it, and is the output usable without the tool? A JSON dump of a proprietary workflow graph is an export and not an exit.

3. What would we have to rebuild? For dbt: orchestration, testing, and the DAG. For Airflow: every DAG, plus whatever business logic drifted into the operators. Writing that list at adoption takes ten minutes and it is never written.

4. And who would notice if we stopped? A tool with no identifiable consumer is one you can leave immediately, and Exercise 25.18's read audit is how you find out.

The discipline that keeps the cost low

Keep the tool thin. The rule that follows from the table above is specific: a DAG should call things, not compute things. Every line of business logic that stays out of an operator is a line you do not have to port, and the rule costs nothing on the day you write it.

The same rule applies to the BI layer and is broken more often. A metric defined in the BI tool is a metric that does not exist in the warehouse, cannot be tested (Chapter 23), and will be redefined by the next tool. Definitions belong in the DAG; the BI tool renders them.

And write the exit plan into the adoption ADR. Not as a plan you intend to execute — as a sentence, in the consequences section, saying what leaving would cost. A tool whose exit cost you cannot state in one sentence is a tool you have not evaluated.

📏 Scale Note — the stack at four team sizes

text 1 engineer 4 (Kestrel) 20 100 ────────────────────────────────────────────────────────────────────────── storage Postgres + object store + a lakehouse the same compute Postgres + DuckDB / + Spark or a + a query a warehouse big warehouse gateway ingestion a cron script 4 mechanisms a vendor tool a self-serve + custom platform transformation SQL files dbt dbt, several dbt + a projects semantic layer orchestration cron Airflow Airflow + a platform team owning it quality nothing dbt tests + a quality + a data SRE + assertions platform function catalog a Markdown a Markdown a catalog a catalog with file file product lineage + a steward systems 1 8 ~14 ~25 per engineer 1.0 1.75 0.7 0.25

Read the last two rows together, because they are the whole callout. Systems grow roughly linearly with team size and systems per engineer falls, which is the actual argument for scale: not that a larger team can run more, but that it can run each thing better.

The column that is hardest is the second, and it is the one most readers are in. Four engineers at 1.75 systems each is under Chapter 5 §5.1's ceiling of 2.0 and there is no slack. Every adoption is a real trade and every one gets made anyway, because each individual decision is reasonable.

Two transitions are worth naming because they are the ones that hurt.

Between 4 and 20, the catalog flips. Not because of table count — because a second team gains write access, and the number of people who can change something without you knowing becomes unbounded (§5.5).

And between 20 and 100, orchestration becomes somebody's job rather than everybody's tool. The failure mode of not making that transition is that the platform's reliability becomes a function of whoever last touched the scheduler, which is Chapter 26's problem with no rotation behind it.

🔐 Privacy & Governance — every tool you adopt is a copy of your data

The systems-per-engineer count in §5.1 has a second reading, and it is the one that arrives two years later as a compliance question: each system is a place personal data lives, and therefore a place it must be deleted from, catalogued in, and access-controlled.

text tool holds personal data? obvious? ──────────────────────────────────────────────────────────── the warehouse yes yes the lake yes yes a search index yes yes a cache yes -- with a TTL sometimes a BI tool's extracts yes NO the orchestrator's logs yes, in task output NO a vector index yes, derived NO a vendor's support system, holding a CSV somebody emailed yes NO

The bottom four rows are where the exposure is, and they share a property: nobody adopted them in order to store personal data. A BI tool caches a query result; an orchestrator logs a task's stdout; an embedding is a numeric derivative. All four are outside every deletion manifest anyone writes by hand, which is Chapter 31's argument for generating it.

Two questions to add to §5.8's evaluation criteria, and they cost a sentence each at adoption:

"Does this system persist data, and for how long?" — including caches, logs, and extracts, and including the answer "we do not control the retention," which is a legitimate answer and a finding.

"How would we delete one person from it?" — asked before adoption, when the answer can still change the decision. Asked afterwards, it is a project.

And the tool-decisions document (Exercise 5.20) gains a column, which is the cheapest possible version of this control: what personal data does this hold, and how does it forget?

🔎 Read the Plan — what docker compose ps tells you that "it's not working" does not

The first debugging skill this book asks for is not about data. It is reading what a stack is actually doing, and it is the skill that determines how much of week one you lose.

text $ docker compose ps NAME IMAGE STATUS PORTS postgres postgres:16 Up 4 minutes (healthy) 5432->5432 minio minio/minio Up 4 minutes (healthy) 9000->9000 redpanda redpandadata/redpanda Up 4 minutes 9092->9092 airflow-ws apache/airflow:2.10.5 Up 3 minutes (unhealthy) 8080->8080 minio-init minio/mc Exited (1) 4 minutes ago

Four facts are in that output and none of them is "it's not working."

minio-init exited non-zero. That is the bucket creation, and it failed. Whatever you were trying to do next was always going to fail, and it is not the thing to debug.

airflow-ws is (unhealthy), not down. The container is running and its health check is failing, which is a different problem from a crash and has a different log.

redpanda has no health status at all — no health check is defined — so "Up" tells you the process started and nothing about whether it works. Exercise 5.19(c) is exactly this distinction: a running container and a working service are different claims.

And two services have been up for four minutes and one for three, which is a startup ordering you can read off the page.

The next command, always:

bash docker compose logs minio-init | tail -20

Read the failing service's log, not the compose output. The compose output is a summary of exit codes; the log is what happened. A student who has read the log has a specific question; one who has not has "it doesn't work" — and the difference is about ten minutes per incident, every time, for the rest of the course.

🔁 Idempotency Check — can you rebuild the stack itself?

The data has to be rebuildable (Chapter 3's 🔁). So does the platform, and the test is the same one: destroy it and bring it back.

bash docker compose down -v # -v removes the VOLUMES. Everything. docker compose up -d python platform/infra/verify_stack.py

Run that once a month. It takes four minutes and it is the only thing that tells you whether the compose file is still the truth.

What it catches, in the order it catches it:

A service that was configured by hand. A bucket created interactively, a database extension installed in a shell, a topic created once and forgotten. All of it survives every restart and none of it survives -v.

A new environment variable that is in your .env and not in .env.example. The rebuild fails, loudly, and the fix is one line in the file that is supposed to be the documentation (Exercise 5.12).

And an ordering assumption that has been holding by luck. Exercise 5.10's race is invisible on a warm machine and reappears on a cold one.

The habit generalises past the sandbox. A production platform's equivalent is Chapter 28's terraform plan on a clean state plus Chapter 34's rebuild from bronze — and a team that has never run either does not know whether it can.

5.9 Summary

"The modern data stack" is a marketing term that implies a package, implies "modern versus legacy," and obscures operational cost. The categories are stable; the products are not. Read every architecture diagram and ask who gets paged when each box breaks — count the systems that can page someone and divide by the team size, and treat roughly two per engineer as the ceiling. Kestrel runs seven systems with four engineers, and that is why a catalog product is not among them.

Eight categories: ingestion, storage, transformation, orchestration, quality, catalog and governance, BI, and reverse ETL. Every platform performs all eight functions, whether or not it has a tool for each.

Buy, build, or do without, and the third is systematically underweighted. The real cost of building is maintenance, not the build. The real cost of buying is integration and exit, not the licence. For a four-person team: buy connectors for SaaS sources and build for your own systems; buy the warehouse and object storage; use dbt; start with dbt tests; do without a catalog and know what that costs; never build a BI tool.

Read the do-without column and note the ranking. Lacking a quality framework is the most expensive gap — you find out from a stakeholder. Lacking a transformation framework compounds fastest. Those two come before anything else; a catalog can wait and reverse ETL can wait indefinitely.

Per-row pricing has a volume ceiling above which it is simply the wrong product. Kestrel's 2.1M monthly changed transactional rows are plausibly worth buying; its 420M monthly clickstream events are not, by orders of magnitude. Real platforms end up with both a bought path and a built path, and budgeting for only one is a planning error.

This book's local stack is honest about its substitutions. PostgreSQL, Kafka, Debezium, dbt, Spark, and Airflow are the real things. MinIO differs from S3 in one line of code. DuckDB is the big leak — single-process, no separated storage and compute, and no cost meter — so cost discipline is taught through arithmetic rather than demonstrated, and Chapter 8 states exactly what changes on a real warehouse.

Six things a small team is told it needs and does not: a streaming platform without an automated consumer, a feature store before models share features, a data mesh before the central team is a bottleneck, an observability product before basic freshness checks, Kubernetes it does not already run, and a semantic layer before metrics are defined in one place.

Below a few hundred gigabytes and ten people, PostgreSQL and a scheduled query is a legitimate architecture — ACID, real constraints, a mature planner, dbt-compatible, operable by one person. The three conditions to move are all observable: analytical load affecting the application, data outgrowing one machine at acceptable cost, or sources PostgreSQL cannot reasonably ingest. Nine months and two engineers once went into replacing a system whose worst query was two seconds slower, because someone reproduced an architecture without re-deriving whether its premises transferred.

Seven evaluation criteria that survive a demo: 3am debuggability, exit cost, price scaling at 10×, who uses it at your scale, what workflow it assumes, whether you can try it with your ugliest data, and the annual maintenance burden visible in its release history. Most people cannot answer the exit-cost or maintenance questions about a tool they depend on daily.

What's next

Chapter 6 is data modeling — dimensional modeling, star schemas, fact grain, and conformed dimensions. It is the oldest material in the book and the most durable: a well-modeled warehouse on mediocre infrastructure beats a badly modeled one on excellent infrastructure, and the gap widens with age. It is also where Kestrel's warehouse gets designed on paper, before a single row moves.