Chapter 5 — Key Takeaways (The Modern Data Stack)
The page for a tooling decision. Note the shelf life: the categories last, the product names do not.
The eight categories
| # | Category | Problem it solves | Kestrel |
|---|---|---|---|
| 1 | Ingestion | Get data in, incrementally, without breaking on schema change | Build (own systems), buy (SaaS) |
| 2 | Storage | Where it lives | Buy — object storage + warehouse |
| 3 | Transformation | Raw → modeled, tested, documented | dbt |
| 4 | Orchestration | Right order, right time, right failure behavior | Airflow |
| 5 | Quality | Is the data right, not just did the job run | dbt tests first |
| 6 | Catalog / governance | What exists, what it means, who can see it | Do without — dbt docs + Markdown |
| 7 | BI | Let people see it | Buy. Never build one. |
| 8 | Reverse ETL | Push data back to operational systems | Build — a script |
The operational ceiling
Count the systems that can page someone. Divide by the number of engineers. Roughly two per engineer, with real coverage on each, is the sustainable ceiling.
Kestrel: 7 systems, 4 engineers. That is why there is no catalog product — a real loss, honestly taken, not a pretense that Markdown is just as good.
Bus factor of one is a problem while the person is still there. They become a bottleneck, and the concentration is self-reinforcing. Two fixes that work and both cost something: rotate on-call and let the second person struggle · require a runbook written by someone who did not set the system up.
Buy / build / do without
The real cost of building is maintenance, not the build. The real cost of buying is integration and exit, not the licence.
| Category | Small team |
|---|---|
| Ingestion — SaaS sources | Buy |
| Ingestion — your own systems | Build (~100 lines you understand) |
| Ingestion — high-volume events | Build (per-row pricing does not apply here) |
| Warehouse, object storage | Buy |
| Transformation, orchestration | Buy (open source) |
| Quality | dbt tests first |
| Catalog | Do without → dbt docs |
| BI | Buy |
| Reverse ETL | Build |
What it costs to do without — ranked
| Missing | Cost |
|---|---|
| Quality framework | 🔴 You find out from a stakeholder. Most expensive gap. |
| Transformation framework | 🔴 Logic scatters into scripts. Compounds fastest. |
| Orchestrator | 🟡 Cron with sleeps. Fails on the day the data is biggest. |
| Ingestion tool | 🟡 Engineering time ∝ source count |
| Warehouse | 🟡 Slower; BI concurrency suffers. Survivable under a TB. |
| Catalog | 🟡 Slow onboarding, duplicated tables, deletion requests become investigations |
| BI tool | 🟢 Analysts share screenshots. Fine at ten people. |
| Reverse ETL | 🟢 Manual CSVs. Embarrassing, and it works. |
Per-row pricing has a volume ceiling. Kestrel: 2.1M monthly transactional rows plausibly worth buying; 420M clickstream events not, by orders of magnitude. Real platforms have both a bought and a built path.
Leave out, until…
| Leave out | Until |
|---|---|
| Streaming platform | Something automated consumes the stream |
| Feature store | Models in production share features |
| Data mesh | The central team is genuinely a bottleneck and domains exist |
| Observability product | You have basic freshness and volume checks |
| Kubernetes | You already run it for something else |
| Semantic layer | Metrics are defined in one place |
The three-way framing (not two-way)
| When | |
|---|---|
| Remove | Another system already covers the capability |
| Narrow | Genuinely needed, far less often than currently used ← systematically overlooked |
| Invest | Essential; the cost is knowledge concentration, not the tool |
Deciding narrow requires measuring actual usage — about a day per system, and it is the day that turns a wrong "remove" into a right "narrow."
This book's substitutions
| Function | Book | Leaks how |
|---|---|---|
| PostgreSQL, Kafka, Debezium, dbt, Spark, Airflow | the real thing | — |
| MinIO → S3 | one line: endpoint_url |
scale, durability, lifecycle policies |
| DuckDB → cloud warehouse | biggest leak | single-process · storage/compute not separated · no cost meter |
DuckDB is superb for SQL semantics, modeling, dbt, incrementals, and plans — all of which transfer.
The Kestrel stack, increment 1
postgres: 16.4, wal_level=logical (set NOW — CDC in Ch. 14 needs it, and
changing it means a restart of a database with data in it)
minio: S3-compatible, 4 buckets: bronze / silver / gold / scratch
minio-init: one-shot, depends_on condition: service_healthy
Four things worth noticing: no password literals (${VAR:?message} fails loudly) · wal_level
set early · health checks with condition: service_healthy (the default waits only for the
container, which is the "works on the second run" bug) · four buckets so access control is a bucket
policy, not a prefix policy.
Three things that go wrong: port 5432 already in use (change the host side) · Docker memory
limit → exit code 137 is an OOM kill, not a crash · stale volumes after a config change →
docker compose down -v deletes data; know that before you need it in a hurry.
When you need none of this
Under a few hundred GB and about ten people: application PostgreSQL → analytics PostgreSQL → scheduled SQL (dbt works fine against it) → BI tool.
ACID, real constraints, a mature planner, one operator, version-controllable.
Move when — all three observable: analytical load affects the application · data outgrows one machine at acceptable cost · sources PostgreSQL cannot reasonably ingest.
Before any platform project: what is our largest table, and how long does our worst query take on the system we already have? Ten minutes. And look at one
EXPLAINbefore accepting that a system is too slow.
Seven criteria that survive a demo
- What happens at 3am? Runbook, logs, real support response time.
- What is the exit cost? ← invisible at purchase, decisive later
- How does the price scale? Model it at 10× today's volume.
- Who uses it at your scale? Not "who uses it." Find someone your size and ask what broke.
- What workflow does it assume? Fighting a tool's opinions is a permanent tax.
- Can you try it with your ugliest data? Every tool works on clean data.
- Annual maintenance burden? Check the release history for breaking changes. Public, and nobody looks.
Most people cannot answer 2 or 7 about a tool they use daily.
Adding a system — the four-part rule
An ADR stating: (a) which existing system was evaluated and why it was insufficient · (b) estimated annual operational hours · (c) who the second operator will be · (d) the condition for removal.
(c) is the one that changes behavior — it moves the knowledge-sharing cost to the front, where it is an afternoon.