Chapter 3 — Key Takeaways (Data Architecture Principles)

The page for a design review. Print it and put it on the table.

The definition

A decision is architectural to the extent that it is expensive to reverse.

It is a spectrum, not a category. Rank your decisions by reversal cost and spend deliberation at the top.

The real work is pushing decisions DOWN this table. Bronze in Parquet turns "which warehouse" from a 12-month decision into a 3-month one. Under uncertainty, optimize for reversibility, not correctness.

Batch vs. streaming

Wrong question: how fresh does it need to be? (Answer is always "as fresh as possible.") Right question: what decision or action changes if this is an hour old instead of a minute old?

daily batch → hourly batch → micro-batch (1–15 min) → streaming (<1s)
simplest, cheapest, easiest to backfill ──────────▶ hardest on every axis

What streaming costs beyond latency: always-on infrastructure · backfills become genuinely hard · state must survive restarts · event time ≠ processing time · you must define a watermark policy to even say what "correct" means · debugging against a moving offset.

At Kestrel's clickstream volume: streaming ≈ $900/month vs. hourly micro-batch ≈ $96/month. ≈9× metered — and the metered part is the smaller cost.

ETL vs. ELT

ELT won because prices changed, not because practice improved. Coupled expensive storage+compute made transform-first rational; $0.023/GB-month storage and elastic compute inverted it.

ETL is still correct in exactly three cases:

1. You legally may not land the raw data (PCI, health identifiers, residency).
2. Volume reduction is enormous and the raw genuinely worthless.
3. The source can be read only once.

Not on the list: "the data is messy," "we don't want junk in the warehouse," "our warehouse is expensive." Those argue for layering, not for discarding evidence.

Warehouse vs. lake vs. lakehouse — by guarantee, not by product

* discipline-dependent — the failure mode is a data swamp

⚠️ A directory of Parquet files is not a table. Two concurrent writers to one S3 prefix interleave silently; a reader that lists mid-write gets a partial result and does not error. That guarantee is exactly what a table format adds.

Three questions that choose: (1) structured with known query patterns? → warehouse. (2) semi-structured, or multiple engines need it? → files in object storage. (3) need transactions, updates, or time travel over those files? → lakehouse.

Kestrel: lakehouse for bronze/silver, warehouse for gold. Not a compromise — each layer using what fits its access pattern.

Sizing

The cloud is not cheaper than owning hardware. It is cheaper than owning hardware sized for your peak. Flat, predictable load may genuinely be better off owned.

$$L = \lambda W \qquad \text{items in system} = \text{arrival rate} \times \text{time in system}$$

At 2,900 events/sec × 40 ms = 116 concurrent handlers needed. Size on the tail (p99), not the mean — the queue builds during slow requests and never drains.

The six surviving principles

1. Choose common components for interoperability and hiring pool, not peak capability.
2. Plan for failure — ask the blast radius and how the system behaves while broken.
3. Architect for scalability in both directions. A system that cannot shrink pays peak prices forever.
4. Build loosely coupled systems. In data, coupling hides in unpublished intermediate tables that quietly became interfaces.
5. Make reversible decisions.
6. Security and cost are design inputs, not afterthoughts. Both are cheap to build in and expensive to add.

The ADR — one page, seven sections

# ADR-NNN: Title
Status / Date / Deciders
## Context            what forced the decision, with numbers
## Decision           what was chosen, in one or two lines
## Alternatives considered    each with WHY IT WAS REJECTED
## Consequences       + and −, honestly
## What would reverse this    ← the most valuable section
## Review date

The three sections people omit: alternatives with reasons, what would reverse this, and a review date.

Reversal conditions must be observable. ✗ "if requirements change" ✓ "if bronze exceeds 2 TB"

Also worth stealing: name the decision most likely to be wrong, inside the document.

Three questions to ask before any storage decision

1. Does any of this data identify a person, directly or in combination?
2. If someone requests erasure, what is the procedure and how long does it take?
3. Is there a retention limit we must honor, and what enforces it?

Ten minutes, asked before the benchmarks. At Kestrel, question 2 eliminated an entire option that cost analysis and performance benchmarks had both favored.