Chapter 3: Data Architecture Principles

"Architecture is the decisions that are hard to change. Everything else is design." — a paraphrase of Martin Fowler, who attributes the underlying idea to Ralph Johnson

Overview

Three arguments dominate data architecture discussion, and all three are usually conducted badly.

Batch versus streaming gets argued as though streaming were the modern option and batch the legacy one, which is roughly like arguing that a helicopter is more modern than a truck. ETL versus ELT gets argued as a matter of best practice when it was settled by a change in what storage costs. Warehouse versus lake versus lakehouse gets argued in vendor terms when the underlying distinction is about guarantees.

This chapter takes each one seriously. That means three things: stating the actual question rather than the tribal position, naming what the losing side costs you, and identifying the specific properties of your situation that decide it. There is no universally correct answer to any of the three, which is precisely why they are still being argued.

Then it does something more useful than settling arguments: it gives you the principles that survive them. The tools in this book will be gone in a decade. The habit of asking what does this decision cost to reverse will not.

One framing to carry through the chapter. Architecture is not a diagram; it is a set of commitments. A diagram with eleven boxes tells you almost nothing. What tells you something is: which of those boxes could be replaced in a sprint, which would take a quarter, and which would take a year. That ranking is the architecture. Everything else is illustration.

In this chapter, you will learn to:

• Ask the real question behind batch versus streaming — which is not "how fast" but "what decision changes" — and apply it to a concrete requirement.
• Explain why ELT displaced ETL as an economic consequence of storage and compute separating, and name the three cases where ETL remains correct.
• Distinguish warehouse, lake, and lakehouse by the guarantees each makes, not by the products that implement them.
• Size a system from peak and average, and compute what each kind of sizing error costs.
• Apply reversible decisions as a ranking tool: classify a choice by what undoing it would cost.
• Write an architecture decision record that a successor can read in four years and understand not just what you chose but what would make them change it.

Who needs this chapter: everyone. It is the chapter people skip when they are in a hurry and regret eighteen months later. On the Quick Start path it is one of the four foundations chapters.

3.1 What Architecture Actually Decides

Start by narrowing the word, because "architecture" is used for everything from a box diagram to a job title.

A decision is architectural to the extent that it is expensive to reverse. That is the whole definition and it is a spectrum rather than a category. It gives you a ranking, and the ranking is more useful than the label.

Here is Kestrel's, ordered by reversal cost:

Two things fall out of this table immediately.

Spend your deliberation budget at the top. An afternoon of argument about the compression codec is an afternoon not spent on the partitioning scheme, and the partitioning scheme is a hundred times more expensive to get wrong. Teams systematically invert this, because the cheap decisions are the ones everyone has an opinion about.

Push decisions down the table when you can. If you can restructure a problem so that an expensive decision becomes a cheap one, you have done real architectural work. Storing bronze data in an open format (Parquet) rather than a proprietary one moves "which warehouse" from a twelve-month decision toward a three-month one, because the data does not have to move. That is the single strongest practical argument for open formats, and it is worth more than any benchmark.

📐 Design Decision — Optimize for reversibility, not for correctness

The instinct is to get the decision right. The better instinct, when you have incomplete information — which is always — is to make the decision cheap to get wrong.

Concretely, at Kestrel: storing bronze in Parquet on object storage rather than loading it straight into the warehouse costs a little more work and one extra hop. What it buys is that changing warehouses does not require re-ingesting from source systems that may no longer have the history. The raw data is yours, in an open format, independent of the query engine.

What you give up: simplicity, and some latency. A single-system architecture — everything in the warehouse — is genuinely easier to operate, has fewer moving parts, and is the right answer for a small enough company. Reversibility is not free, and buying it when you do not need it is its own mistake.

The test: how confident are you in this decision, and how long will you be living with it? At high confidence and short duration, optimize for simplicity. At low confidence and long duration — which describes most early architecture — buy reversibility.

3.2 Batch versus Streaming

The question people ask, and the question that decides it

The question people ask is how fresh does the data need to be? Asked that way, the answer is always "as fresh as possible," because nobody has ever asked for staler data.

The question that decides it is: what decision or action changes if this data is one hour old rather than one minute old?

That reframing does most of the work. In Chapter 1's Case Study 1, ten weeks went into a streaming pipeline for a growth team that was making weekly decisions on Monday mornings. Nobody was lying; the growth team genuinely wanted fresh data, in the same way that everyone genuinely wants a faster laptop. The follow-up question was never asked.

Answers cluster into three groups, and only the third is genuinely streaming:

"Nothing changes, but it feels better." By a wide margin the most common. Build a batch job. If the anxiety is real, run it more often — hourly batch satisfies almost everyone who asks for real-time.

"I would catch a problem sooner." Legitimate, and almost always satisfied by monitoring and alerting rather than by a streaming data pipeline. If someone wants real-time inventory data because they are worried about stockouts, what they want is a stockout alert, not a stream. Chapter 25.

"An automated system acts on it without a human." Fraud scoring, inventory allocation, dynamic pricing, personalization, operational alerting. This is genuine streaming, and Chapter 29 is written for it. The distinguishing feature is that the latency requirement comes from a machine's response time rather than a human's attention span.

The spectrum, honestly

Batch and streaming are not a binary. There is a continuous spectrum and most useful systems live in the middle:

  daily batch      hourly batch     micro-batch      streaming
  ┌──────────┐     ┌──────────┐     ┌──────────┐    ┌──────────┐
  │  24 h    │     │   1 h    │     │  1-15min │    │  <1 sec  │
  └──────────┘     └──────────┘     └──────────┘    └──────────┘
  simplest ───────────────────────────────────────▶ hardest to operate
  cheapest ───────────────────────────────────────▶ most expensive
  easiest to reason about ────────────────────────▶ hardest to reason about
  easiest to backfill ────────────────────────────▶ hardest to backfill

In words: as you move right, latency improves and every operational property gets worse. The middle of the spectrum — micro-batch, one to fifteen minutes — is where a surprising amount of "real-time" demand is genuinely satisfied, and it is operationally much closer to batch than to streaming.

What streaming actually costs

The latency benefit is obvious and gets all the attention. The costs are less obvious and get almost none, so here they are explicitly:

Always-on infrastructure. A batch job runs and stops. A stream consumer runs forever, which means it must be monitored forever, restarted, upgraded in place, and kept from falling behind. This is the largest cost and it is entirely operational.

Backfills become genuinely hard. Re-running a batch job for last Tuesday is a parameter change. Re-processing a stream for last Tuesday means replaying from a retained offset, into a state store that may have moved on, without double-counting the output. Chapter 29 §29.7 is honest about how unpleasant this is.

State management. Aggregations over a window need somewhere to keep partial results. That state must survive restarts and rebalances. It is the source of most streaming bugs.

Time becomes ambiguous. Event time is when it happened; processing time is when you saw it. They differ, and mobile clients that go offline make them differ by hours. Every window computation has to choose, and the naive choice — processing time — is wrong in a way that is invisible until a mobile client reconnects. Chapter 29 §29.4.

Correctness gets harder to define. In batch, "correct" means "matches what a full recomputation would produce." In streaming, with late data and windows, you need a watermark policy to even define what the right answer is, and the policy is a business decision disguised as a technical one.

Debugging gets harder. A batch job's input is a file you can look at. A stream's input is a moving offset, and by the time you investigate, the messages may have aged out of retention.

💸 Cost Check — What always-on costs at Kestrel's clickstream volume

Kestrel's clickstream is 14M events/day, 162/sec average, 2,900/sec peak.

Streaming. Consumers must be sized for peak, because falling behind at peak is the whole failure mode. Say three consumer instances plus a three-broker Kafka cluster, running 24×7. At the frozen managed-Kafka rate of $0.21/broker-hour:

$$3 \text{ brokers} \times 24 \times 30 \times \$0.21 = \$453.60 \text{ per month, brokers alone}$$

Plus the consumers, plus storage, plus the state store. Call it roughly $900/month all in, and it is $900 whether the events are flowing or not.

Hourly micro-batch. Read the last hour's events, aggregate, write. Twenty-four runs a day at roughly four minutes each on a small warehouse. On Snowflake at the frozen $2.00/credit with an X-Small at 1 credit/hour:

$$24 \times \frac{4}{60} \times 1 \times \$2.00 \times 30 = \$96.00 \text{ per month}$$

Roughly a 9× difference in metered cost — and the metered cost is again the small part. The streaming version needs a person who understands consumer groups, rebalancing, and state stores, available when it breaks. At a four-person team, that is the binding constraint.

This arithmetic does not mean streaming is wrong. It means streaming should be paid for by a requirement that names a machine acting on the data, not by a preference.

🎓 Interview Angle — "How would you decide between batch and streaming?"

The answer that gets an offer is not a list of trade-offs; the interviewer has heard the list. It is an answer that starts by refusing the question as asked:

"First I'd want to know what acts on the data and how fast it has to act. If it's a human looking at a dashboard, I'd start with batch or micro-batch and make the case that hourly is almost certainly sufficient — and I'd want to be wrong about that in a cheap way, so I'd design the batch job so the same transformation logic could run in a stream later. If it's an automated system — fraud, allocation, pricing — then it's genuinely streaming and I'd want to talk about delivery semantics and event time versus processing time, because that's where the correctness bugs are."

Then name the operational cost. Candidates who mention that a stream is always-on and that backfills are hard are demonstrating that they have operated one.

3.3 ETL versus ELT

What changed, and why it was economics

ETL — extract, transform, load — transforms data before it lands in the warehouse. ELT loads raw data first and transforms it inside the warehouse.

The industry moved from the first to the second, and it is important to understand that this was not a discovery about better practice. It was a response to a change in prices.

In the ETL era, a warehouse was an appliance you bought. Storage and compute were bought together, in a fixed quantity, at high cost. Every gigabyte you loaded consumed capacity you had paid for up front. Under those conditions, loading raw data you were not going to query was economically irrational, so you cleaned and aggregated first, on cheaper general-purpose hardware, and loaded only the result.

In the ELT era, S3 made storage effectively unlimited at $0.023/GB-month, and Snowflake and BigQuery made compute elastic and billed by the second. Once storage is nearly free and compute is on demand, the calculus inverts. Loading raw data costs almost nothing, and having it is worth a great deal:

• You can answer questions you did not anticipate, without re-extracting from a source system that may no longer have the history.
• You can reprocess when you find a bug in the transformation — and you will — instead of discovering that the correct data was discarded in 2023.
• You can debug, because the first question in every investigation is "what did the source actually say," and the raw layer is the only thing that answers it.
• Transformation happens in SQL, which vastly more people can read, review, and modify than whatever the ETL tool's scripting language was.

The picture

   ETL                                  ELT
   ┌────────┐                           ┌────────┐
   │ source │                           │ source │
   └───┬────┘                           └───┬────┘
       │ extract                            │ extract
       ▼                                    ▼
   ┌───────────────┐                    ┌────────────┐
   │  transform    │  <- separate       │  load raw  │
   │  (ETL server) │     compute        │  (bronze)  │
   └───┬───────────┘                    └───┬────────┘
       │ load                               │ transform  <- in the warehouse,
       ▼                                    ▼               in SQL
   ┌───────────────┐                    ┌────────────┐
   │  warehouse    │                    │ silver/gold│
   │  (clean only) │                    └────────────┘
   └───────────────┘
   raw data is GONE                     raw data is KEPT

In words: ETL transforms on separate compute and loads only the result, so the raw data is not retained. ELT loads raw data into cheap storage first and transforms inside the warehouse, so the raw layer remains available for reprocessing and investigation.

Where ETL is still right

The book's default is ELT and it will be used throughout. But three cases genuinely call for transforming before loading, and an engineer who does not know them will build the wrong thing when one arrives.

1. When you legally may not land the raw data. If a source contains personal data you are not permitted to store — full payment card numbers, health identifiers outside a compliant boundary, data subject to a residency requirement — you cannot land it raw and clean it later. The transformation (tokenization, redaction, filtering) must happen before it touches your storage. This is not negotiable and it is the strongest case for ETL. Chapter 31 §31.5.

2. When the volume reduction is enormous and the raw data is genuinely worthless. If a sensor emits 10,000 readings per second and you only ever need per-minute averages, landing all of it costs real money for data nobody will query. Be careful with this argument — "we'll never need the raw data" has an extremely poor track record — but at some volume ratio it becomes true.

3. When the source can only be read once. Some legacy systems, some queue-based interfaces, and some vendor APIs give you one shot. If parsing is required to make the data usable at all and you cannot go back, you transform in flight.

Note what is not on this list: "the raw data is messy," "we don't want junk in the warehouse," and "our warehouse is expensive." The first two are arguments for good layering, not for discarding evidence. The third is an argument for putting bronze in object storage, which is exactly what the medallion architecture does.

📏 Scale Note — Where ELT's economics break down

ELT's premise is that storage is cheap relative to the value of keeping raw data. At Kestrel that is overwhelmingly true: the full raw clickstream is 4.19 TB/year, costing $96.37/month at the frozen S3 rate.

At 1,000× Kestrel — a large ad-tech or telemetry business, 14 billion events a day — the same arithmetic gives 4.19 PB/year and roughly $96,000/month for one year of raw retention. That is a real budget line and it changes the decision.

What large-volume shops actually do is neither pure ETL nor pure ELT: they land raw with a short retention (7 to 30 days), transform to a compact form, and retain the transformed data long-term. You keep reprocessing ability for the window in which bugs are actually found, and you pay long-term storage only for data you will query. The retention window becomes an explicit, priced architectural parameter rather than a default.

Kestrel's answer: bronze retained two years, because $96/month makes the question uninteresting. Notice that the right answer changed because a price changed, not because a principle did.

3.4 Warehouse versus Lake versus Lakehouse

These three terms are used loosely, including by vendors who benefit from the looseness. Define them by guarantees rather than by products and the distinctions become sharp.

Data warehouse

What it is: a system that stores structured data in a schema optimized for analytical queries, with a query engine tightly integrated with the storage.

What it guarantees: schema on write (data that does not fit is rejected at load time); ACID transactions; strong consistency; SQL access; a query optimizer with statistics; and access control at table, column, and row level.

What it costs: you must define the schema up front. Semi-structured and unstructured data fit badly or not at all. Storage is more expensive than object storage. Historically you were locked into the vendor's format, though this is eroding fast as warehouses learn to read Iceberg.

Examples: Snowflake, BigQuery, Redshift, Databricks SQL warehouse, and — at Kestrel's local scale — DuckDB.

Data lake

What it is: files in object storage, in open formats, with a query engine pointed at them.

What it guarantees: almost nothing, and that is the point. Any format, any structure, any volume. Cheapest possible storage. Complete decoupling of storage from every engine that reads it.

What it costs: no transactions, no schema enforcement, no consistency guarantee across files, no update or delete without rewriting whole files, and no built-in metadata. A reader who arrives mid-write sees a partial result. Two writers to the same prefix corrupt each other silently.

The failure mode has a name — the data swamp — and it is not caused by the technology. It is caused by the absence of the discipline the warehouse used to enforce for you. Nobody knows what is in it, what any file means, which is current, or who owns it.

Examples: Parquet on S3, queried with Athena, Spark, Trino, or DuckDB.

Lakehouse

What it is: a table format layered over files in object storage that adds a transaction log, giving lake economics with warehouse-like guarantees.

What it guarantees: ACID transactions over object storage; schema enforcement and evolution; time travel (query the table as of a past version); efficient upserts and deletes; and open formats readable by many engines.

What it costs: more complexity than plain files. A metadata layer to maintain. Compaction and vacuum as ongoing operations rather than optional tidiness. And a genuinely young ecosystem — this is the most actively changing area in the field, and interoperability between engines and table formats is uneven and improving.

Examples: Delta Lake, Apache Iceberg, Apache Hudi.

The comparison that matters

⚠️ Failure Mode — The concurrent write that corrupted a partition

The single most consequential row in that table is ACID transactions: No for a plain data lake, and it is worth making concrete.

A Spark job writes s3://kestrel/bronze/events/event_date=2025-06-11/. It writes 240 part files over eleven minutes. At minute six, a second job — a backfill someone triggered manually — starts writing the same prefix.

There is no lock. There is no transaction. Object storage cheerfully accepts both. The result is a prefix containing an interleaved mixture of two jobs' output, with duplicated rows where they overlapped and, because the first job's _SUCCESS marker lands before the second finishes, nothing anywhere indicates a problem.

Meanwhile a query that started at minute four sees whatever files existed at the moment it listed the prefix — a partial write. It returns fewer rows than it should. It does not error.

This is not a bug in S3. Object storage is doing exactly what it promises: storing objects. The guarantee that a set of files constitutes a consistent table version is precisely what a table format adds, and it is why Chapter 10 exists. If you take one thing from this section, take this: a directory of Parquet files is not a table. It is a directory of Parquet files, and treating it as a table works right up until two writers meet.

How to choose

Rather than a flowchart, three questions that actually decide it:

1. Is your data structured and your query pattern known? If yes, and it fits, a warehouse is simpler and faster and you should probably stop there. Most companies under a few terabytes should.

2. Do you have semi-structured or unstructured data, or multiple engines that need the same data? Then you need open files in object storage, and the question is only whether you add a table format on top.

3. Do you need transactions, updates, or time travel over those files? If yes, lakehouse. If you are genuinely append-only with a single writer and you can enforce that socially, a plain lake is simpler and you may not need the metadata layer.

Kestrel's answer, recorded properly in §3.7: a lakehouse for bronze and silver (Delta over object storage, because the clickstream is semi-structured, high-volume, and needs deletes for privacy requests), and a warehouse for gold (DuckDB locally, Snowflake in the cloud version, because that is what BI tools and analysts want to query). This hybrid is extremely common and it is not a compromise — it is each layer using the technology that fits its access pattern.

3.5 Sizing: Peak, Average, and the Cost of Being Wrong

Kestrel's peak-to-average ratio for orders is 6.28× and for clickstream events is roughly 18×. Those two numbers drive more architecture than the annual volumes do, and this section is the arithmetic.

The two ways to be wrong

Size for the average and you fail at peak. Black Friday brings 41,300 orders against a 6,575 average. A pipeline sized for the average, degrading roughly linearly, takes about 6.28× as long — a 22-minute job becomes 138 minutes, and the 6am SLA is missed on the single day of the year when the business is watching most closely.

Size for the peak and you pay for idle capacity 364 days a year. Provision for 41,300 orders and you are running at 16% utilization on a typical day.

💸 Cost Check — What each sizing error costs, on one basis

Say the nightly transformation needs 8 nodes at average volume and 48 at Black Friday volume, using the frozen rate of $2.400/node-hour and a 1.5-hour run.

Fixed at peak (48 nodes, always): $$48 \times 1.5 \times \$2.400 \times 365 = \$63{,}072 \text{ per year}$$

Fixed at average (8 nodes, always): $$8 \times 1.5 \times \$2.400 \times 365 = \$10{,}512 \text{ per year}$$ — and the 6am SLA is missed on the highest-traffic days of the year.

Elastic (8 nodes for 355 days, 48 for the 10 peak days): $$(8 \times 355 + 48 \times 10) \times 1.5 \times \$2.400 = \$11{,}952 \text{ per year}$$

Elastic costs $1,440 more than under-provisioning and $51,120 less than over-provisioning — 19.0% of the fixed-at-peak bill. And it meets the SLA every day.

This is the entire economic argument for cloud infrastructure in a business with a peak, and it is worth being precise about it: the cloud is not cheaper than owning hardware. It is cheaper than owning hardware sized for your peak. A business with flat load and predictable growth may genuinely be better off on its own machines, and some are.

Little's Law, which is the only formula in this chapter

For any stable system:

$$L = \lambda W$$

where $L$ is the average number of items in the system, $\lambda$ is the arrival rate, and $W$ is the average time an item spends inside. It is exact, it requires almost no assumptions, and it is the fastest way to sanity-check a capacity plan.

Substituting Kestrel's clickstream at peak: events arrive at $\lambda = 2{,}900$ per second, and suppose your consumer takes $W = 0.04$ seconds to handle one:

$$L = 2{,}900 \times 0.04 = 116$$

You need 116 events in flight concurrently to keep up. If your consumer processes one at a time with no parallelism, you need 116 consumer threads, or you fall behind — and falling behind at 2,900/sec compounds fast. Twelve partitions with ten concurrent handlers each gives you 120. That is where the "12 partitions" in Kestrel's topic configuration comes from; it is not a round number someone liked.

🧪 Try It — Size the consumer yourself

Using $L = \lambda W$:

1. Your consumer's median handling time is 12 ms. At the 2,900/sec peak, how many concurrent handlers do you need?
2. Kestrel's topic has 12 partitions, and a Kafka partition is consumed by at most one consumer in a group. How many concurrent handlers per partition does your answer require?
3. Now suppose handling time has a long tail: p50 is 12 ms but p99 is 400 ms. Recompute using p99. What does the gap between the two answers tell you about sizing on averages?

The third question is the point. Sizing on the mean is how systems fall over at peak, because the queue builds during the slow requests and never drains.

3.6 The Principles That Survive

Tools change. These do not. Six principles, each with the specific way it shows up in this book.

1. Choose common components wisely. Shared components — object storage, the orchestrator, the warehouse — are used by everyone and therefore constrain everyone. Choose them for interoperability and for the size of the pool of people who know them, not for peak capability. An exotic tool that is 20% better and that nobody you can hire has used is a bad trade.

2. Plan for failure. Not "if" but "what happens when." Every component fails: the source database, the network, the object store, the orchestrator, the warehouse, and you. Chapter 26 turns this into concrete practice. The design question is always what is the blast radius, and how does the system behave while it is broken?

3. Architect for scalability — and know which direction. Scaling up is a bigger machine; scaling out is more machines. Scaling down matters as much and is more often forgotten: a system that cannot shrink is a system paying peak prices continuously, which is exactly the $51,120 in §3.5.

4. Build loosely coupled systems. Coupling is what makes a change in one place break something in another. In data platforms coupling hides in specific places: a downstream job that reads another job's internal table; a transformation that depends on a column's physical position; a consumer that assumes an upstream's file layout. The mechanism for decoupling is an explicit interface — which in data means a contract (Chapter 17) and a published table with a documented grain, not an implementation detail someone else discovered.

5. Make reversible decisions. §3.1. Given uncertainty, prefer the choice that is cheaper to undo, and prefer designs that convert expensive decisions into cheap ones.

6. Prioritize security and cost as design inputs, not afterthoughts. Both are undercurrents (Chapter 2 §2.7). Both are cheap to build in and expensive to add. A platform that never modeled where PII lives cannot honor a deletion request without an investigation; a platform that never attributed cost cannot answer which pipeline to optimize.

🏭 From the Pipeline — Coupling nobody could see

A reporting job at a retailer read from staging.orders_tmp, an intermediate table produced by the loader, rather than from the published gold.fct_order. It had been written that way years earlier because orders_tmp had a column that had not yet been promoted to the fact table, and the reason was long forgotten.

When the loader was rewritten to use a different intermediate structure, orders_tmp disappeared. The reporting job broke, and — because it was a weekly job — it broke silently for six days before anyone noticed.

Nothing in the codebase indicated the dependency. Lineage tooling would have caught it, and there was none. The published fact table had a contract; the intermediate table was an implementation detail that had quietly become an interface.

The general rule: if it is not published, it is not an interface, and if people are reading it anyway you have coupling you cannot see. Two defenses — access control that prevents reading intermediate schemas at all, and lineage that shows you who reads what (Chapter 30 §30.4).

3.7 Kestrel's Architecture Decision Record

A decision without a written record is a decision your successor will re-litigate from scratch in two years, badly, because they will not know what you rejected or why.

An architecture decision record (ADR) is a short document — one page — with a fixed structure. Here is Kestrel's first one, complete, as a model.

# ADR-001: Lakehouse for bronze/silver, warehouse for gold

Status:   Accepted
Date:     2025-01-14
Deciders: data engineering (4), with review from analytics and security

## Context

Kestrel ingests from three sources: a 340 GB PostgreSQL OLTP database, a
clickstream at 14M events/day (4.19 TB/year raw), and several third-party
APIs. Consumers are BI (22 dashboards), two data scientists, and a finance
reconciliation. The team is four engineers. The binding SLA is that the
daily_revenue dashboard is fresh for the prior day by 06:00 America/New_York.

Two properties drive the decision:
  - The clickstream is semi-structured, high-volume, and schema-unstable at
    the client.
  - Privacy requests (Chapter 31) require deleting individual rows from
    historical data, including data older than any warehouse retention we
    would want to pay for.

## Decision

  bronze + silver : Delta Lake tables on object storage
  gold            : warehouse (DuckDB locally; Snowflake in the cloud build)

## Alternatives considered

1. Everything in the warehouse. Simplest to operate, one system, one access
   model. REJECTED: 4.19 TB/year of raw clickstream in warehouse storage is
   materially more expensive than object storage, semi-structured events fit
   badly, and we would lose the raw layer we need for reprocessing.

2. Everything in a plain data lake (Parquet, no table format). Cheapest, and
   fully engine-independent. REJECTED: no ACID means concurrent writers can
   corrupt a partition silently (see the incident class in Chapter 3 section
   3.4), and row-level deletes for privacy requests would require rewriting
   whole partitions by hand.

3. Iceberg instead of Delta. Genuinely close. Iceberg has broader engine
   support and a cleaner catalog story. CHOSE DELTA because our Spark and
   local tooling support is better today and the team has more experience
   with it. THIS IS THE DECISION MOST LIKELY TO BE REVISITED — see below.

## Consequences

  + Cheap raw retention (bronze: 2 years at ~$96/month for clickstream)
  + Row-level deletes for privacy requests without rewriting partitions
  + Time travel for debugging and reproducible reprocessing
  + Gold stays in the system analysts and BI tools expect
  - Two storage systems to operate, monitor, and secure
  - Compaction and vacuum become standing operational duties
  - Engineers must understand both models

## What would reverse this

  - If total analytical data stays under ~2 TB for two years, collapse to
    warehouse-only. The complexity would no longer be earning its keep.
  - If Iceberg support in our engines clearly surpasses Delta, migrate the
    table format. Cost estimated at 3-4 weeks; the data itself is Parquet
    either way, which is why this is a weeks decision and not a quarters one.
  - If we adopt a second processing engine that reads Iceberg better than
    Delta, revisit immediately.

## Review date

  2026-01-14, or on any of the reversal conditions above.

Three things make this ADR worth its page, and they are the three most often missing:

The alternatives, with reasons for rejection. Without them, a successor cannot tell whether you considered their idea and rejected it or never thought of it.

"What would reverse this." The single most valuable section, and almost nobody writes it. It converts a decision from a permanent commitment into a conditional one, and it tells your successor exactly what evidence to look for.

A review date. Decisions expire. This one names when to look again.

🧱 Kestrel Platform — Increment 3: your first ADRs

Create platform/docs/adr/ and write two records using the template above.

ADR-001 — write your own version of the storage decision. Do not copy the one above; work through it with the numbers from Chapter 1 §1.5 and reach your own conclusion. If you conclude warehouse-only, that is a legitimate answer for Kestrel and you should say what would change it.

ADR-002 — batch versus streaming for the clickstream. Apply the §3.2 test explicitly: name what acts on the data and how fast it must act. State the cost of each option using the frozen basis.

Then the part that matters most: for each, fill in "What would reverse this" with at least two specific, observable conditions. Not "if requirements change" — something you could write a monitoring query for.

Chapter 38 asks you to reread both and mark which reversal conditions have been met.

Build, buy, or do without

Every architectural decision in this chapter has a fourth option that the framing tends to hide. The question is usually posed as "which tool", and the prior question is "do we operate a tool at all."

Three options, and the third is real:

BUY          somebody else operates it. You pay money and lose control.
BUILD        you operate it. You pay engineering and gain control you
             usually do not use.
DO WITHOUT   you accept the gap, deliberately, and write down what you
             gave up.

"Do without" is a legitimate architecture and it is the one nobody writes an ADR for, which is exactly why it decays into an accident. Kestrel has no data catalog (Chapter 5 §5.5); the decision is correct at four engineers and it is only defensible because there is a written threshold at which it changes.

The three questions that decide it

1. Is this our differentiator? Almost nothing in a data platform is. A company that sells outdoor gear does not differentiate on its orchestrator. Build only where the thing you build is the product, or where the requirement is genuinely unusual enough that no product addresses it — and "unusual" needs evidence, because every team believes its requirements are unusual.

2. What is the total operational cost, not the licence? Chapter 5 §5.1's systems-per-engineer count is the honest accounting. A self-hosted tool costs: the machines, the upgrades, the incidents, the on-call knowledge, the second operator you must be able to name, and the fact that its failure modes are now yours to learn.

3. What is the exit cost, and have you priced it? This is the question that separates a decision from a purchase. Buying is reversible in proportion to how much of your logic lives inside the tool — a managed Postgres is nearly free to leave and a workflow tool whose business rules live in a GUI is nearly impossible (Chapter 37 §37.12).

💸 Cost Check — the total cost of a "free" self-hosted tool

A team proposes self-hosting an orchestrator rather than paying for a managed one at ~$350 a month.

text the licence $0 two instances (prod + non-prod), plus a metadata DB ~$310 / month upgrades: 3/year x 1 day, incl. testing ~3 days / year incidents attributable to the tool: ~4/year x 0.5 d ~2 days / year the learning curve, one engineer, ongoing ~4 days / year ───────────── ~9 engineer-days + $3,720 / year

Nine engineer-days is roughly $6,000–$10,000 fully loaded, so the "free" option costs about $10,000–$14,000 a year against $4,200 for the managed one — and the managed one does not consume a quarter of an engineer's attention at unpredictable times.

The trap in this arithmetic is that the engineering days are invisible. They arrive as "we had an upgrade this sprint," which nobody attributes to a build-versus-buy decision made two years earlier. The licence is the only line anybody puts in a spreadsheet, and it is the smallest one.

When self-hosting is right: when you already run the substrate (you have Kubernetes and a team that operates it), when the managed option's cost scales with something you have a lot of, or when a compliance requirement genuinely forbids it. All three are checkable claims, and an ADR should contain the check rather than the assertion.

🔐 Privacy & Governance — three architecture decisions that are privacy decisions

These are made in Chapter 3 and paid for in Chapter 31, and none of them looks like a privacy decision at the time.

Partitioning. An erasure request against an unpartitioned table rewrites every file that contains the person. Partitioned by date, it rewrites the partitions they appear in. The partitioning scheme decides what a deletion costs, and it is chosen for query performance by someone not thinking about deletion.

Retention. "Keep everything" is not a neutral default. Every year retained is a year of erasure obligation, a year of breach surface, and a year of data that must be included in a subject access request. ADR-003's two-versus-90-days decision is a privacy decision with a storage cost attached, not the reverse.

Where the boundary sits between systems. Every copy of personal data is a copy that must be deleted, catalogued, and access-controlled. A hybrid architecture (§3.6) has more copies than a single warehouse, and the extra copies are the ones nobody lists — a search index, a cache, a feature store, an embedding.

The practical form: add one line to the ADR template. "What personal data does this decision cause us to hold, where, and for how long?" It takes a sentence to answer and it is the difference between a privacy programme that inspects decisions and one that discovers them.

🧭 Version Note — the architectures that dated, and the one that did not

A short history, because you will meet all of these in job descriptions and in other people's systems.

text era the shape what killed it ───────────────────────────────────────────────────────────────────────── ~1995 EDW: one warehouse, ETL, cost, and the modelling everything modelled up front bottleneck ~2010 Hadoop: HDFS + MapReduce, operational weight; nobody "schema on read" could query it ~2015 Lambda: batch + speed layers the logic was written twice ~2016 Kappa: everything is a stream replaying history at any real scale is not an operation ~2018 the "modern data stack": nothing yet; but it assumes ELT into a cloud warehouse your data fits a warehouse ~2020 the lakehouse: a transaction nothing yet log over object storage

The one that did not date is dimensional modelling (Chapter 6), which is from 1996 and is unchanged. That is not an accident. Every row above is a statement about where computation happens and what it costs, and those change every five years. Dimensional modelling is a statement about what the data means, which does not.

The reading rule that follows: when you meet older architectural material, separate its claims about cost from its claims about meaning. The cost claims are almost certainly stale; the meaning claims are probably still right. Hadoop-era advice about avoiding joins is a cost claim and it is obsolete; Kimball-era advice about declaring the grain is a meaning claim and it is not.

And the corollary for your own ADRs: date the cost assumptions explicitly. "At $0.023/GB-month, retaining two years costs $2,028/year" survives being read in 2031, because a future reader can see which number to update. "Storage is cheap" does not.

🔁 Idempotency Check — the architectural version

Idempotency is usually taught as a property of a write. It is also a property of an architecture, and the architectural form is: can this system be rebuilt from its inputs?

text a lake with retained bronze YES -- Chapter 34's $198.96 a warehouse loaded by an ETL job that transformed before landing NO -- the inputs are gone a stream with 7-day retention for 7 days a system of record by definition, no

Every architecture in this chapter has an answer to that question, and the answer is a design property rather than an operational one. A lakehouse can be rebuilt because bronze exists; a classical ETL warehouse cannot, because the transformation happened before anything was stored (§3.3).

This is worth adding to the ADR template as a one-line question, because it is cheap to answer at decision time and impossible to retrofit: "if we had to rebuild this from scratch, what would we rebuild it from, and have we ever tried?"

3.8 Summary

Architecture is the set of decisions that are expensive to reverse, and the useful move is to rank your decisions by reversal cost rather than to label some of them architectural. Kestrel's ranking runs from cloud provider (1–2 years) to BI tool (days). Spend deliberation at the top, and do real architectural work by pushing decisions down the ranking — storing bronze in an open format turns "which warehouse" from a twelve-month decision into a three-month one.

Batch versus streaming is decided by what acts on the data, not by how fresh anyone wants it. Three answers to "what changes if it's an hour old": nothing (build batch), I'd catch problems sooner (build monitoring), an automated system acts on it (build streaming). Only the third pays for the costs, which are always-on infrastructure, hard backfills, state management, event-time ambiguity, a watermark policy you must define, and debugging against a moving offset. At Kestrel's volume the metered difference is roughly 9× — and the metered cost is the smaller part.

ELT displaced ETL because prices changed, not because practice improved. When storage and compute were coupled and expensive, transforming first was rational. When storage became $0.023/GB-month and compute became elastic, keeping raw data became nearly free and worth a lot. ETL is still correct in exactly three cases: when you legally may not land the raw data, when volume reduction is enormous and the raw genuinely worthless, and when the source can only be read once. "Messy data" is not on that list — it is an argument for layering, not for discarding evidence.

Warehouse, lake, and lakehouse are distinguished by guarantees. A warehouse gives schema on write, ACID, and an optimizer, at higher storage cost and poor fit for semi-structured data. A lake gives cheapest storage and total engine independence, and guarantees essentially nothing — a directory of Parquet files is not a table, and that stops being an abstraction the first time two writers meet in the same prefix. A lakehouse adds a transaction log over lake files, buying ACID, schema evolution, time travel, and row-level deletes, at the cost of the highest operational burden of the three and the youngest ecosystem.

Sizing is where the peak-to-average ratio earns its place as the most architecturally important number. At Kestrel, fixed-at-peak costs $63,072/year, fixed-at-average costs $10,512 and misses the SLA on the days that matter, and elastic costs $11,952 — 19.0% of fixed-at-peak, meeting the SLA every day. That is the actual economic argument for cloud infrastructure: not that it is cheaper than hardware, but that it is cheaper than hardware sized for your peak. And $L = \lambda W$ is the fastest capacity sanity check there is — size on the tail, not the mean, because the queue builds during slow requests and never drains.

Six principles survive the tools: choose common components for interoperability and hiring pool; plan for failure by asking what the blast radius is; architect for scalability in both directions, including down; build loosely coupled systems, where coupling in data platforms hides in unpublished intermediate tables that quietly became interfaces; make reversible decisions; and treat security and cost as design inputs rather than afterthoughts.

Write the decision down. An ADR needs three things people usually omit: the alternatives with reasons for rejection, a review date, and — the most valuable section in the document — what would reverse this, stated as conditions specific enough that you could write a monitoring query for them.

What's next

Chapter 4 is the theory chapter, and it is short because it is aggressively selective: partitioning, replication, consistency, and delivery semantics, taken only as far as you need to debug a real pipeline. Every hard bug in this field is a distributed systems bug wearing a costume, and Chapter 4 is how you learn to recognize the costume — including the reason "exactly-once" is usually a claim about one hop rather than about your system.