Case Study 2: Choosing the Storage Architecture

"The decision took two weeks. Writing down why took two hours, and it is the only part anyone has read since."

Executive Summary

In January 2025, Kestrel's data team had to decide where its analytical data would live. Three options were genuinely on the table: everything in a warehouse, everything in a plain data lake, or a hybrid lakehouse-plus-warehouse.

This case study is the decision, worked properly — the arguments on each side, the arithmetic, the two considerations that turned out to decide it, and the ADR that resulted. It then does something a case study usually cannot: it revisits the decision eighteen months later against the reversal conditions the ADR wrote down, and reports which ones fired.

The conclusion Kestrel reached is the one Chapter 3 §3.4 states. The value here is entirely in the method — specifically in how two of the three options were eliminated by considerations that were not on anyone's list when the discussion started.

Skills applied: warehouse/lake/lakehouse guarantees (§3.4); reversal cost (§3.1); ADR structure and reversal conditions (§3.7); cost arithmetic on the frozen basis.

Background

The situation, January 2025. Four data engineers. Three sources: a 340 GB PostgreSQL OLTP database, a clickstream at 14M events/day, and several third-party APIs. Consumers: 22 dashboards (after the consolidation described in Chapter 2's Case Study 2), two data scientists, and a monthly finance reconciliation.

The existing arrangement was accidental rather than designed — the legacy of the first data hire's "embarrassingly simple" Python-script-to-Postgres-schema system, extended repeatedly. It had reached the end of what extension could do. Clickstream data had never been properly landed at all; it was being sampled at 5% and aggregated in flight, and the data scientists had been asking for the full stream for a year.

The forcing function was a privacy requirement. Kestrel had begun selling in the UK, and the legal team had asked a question the platform could not answer: if a customer requests erasure, can we delete their data everywhere, and how long does it take? The honest answer was "we do not know, and probably weeks."

That question turned out to decide the architecture, though nobody expected it to when the discussion started.

The Problem

Three options, each with a genuine constituency on the team.

Option A — everything in the warehouse

Land raw into warehouse staging tables, transform in SQL, serve from the same system.

For: one system to operate, secure, monitor, and back up. One access model. One query language. One vendor to call. For a four-person team, operational simplicity is not a soft benefit — it is the binding constraint, and this option is dramatically the simplest.

Against: the clickstream. 4.19 TB/year of semi-structured JSON events, with a schema that changes whenever a front-end team ships. Warehouses handle semi-structured data — Snowflake's VARIANT, BigQuery's JSON — but handle it as a second-class citizen, and warehouse storage costs materially more than object storage.

Option B — everything in a plain data lake

Parquet files on object storage. Query with DuckDB locally, Trino or Athena in the cloud. No warehouse at all.

For: cheapest storage by a wide margin. Total engine independence — any tool that reads Parquet can read your data, forever, with no vendor in the path. Handles any data shape.

Against: no ACID, no schema enforcement, no updates or deletes without rewriting whole files. And the thing the analysts cared about: BI tools connect to warehouses well and to file-based query engines less well, with worse concurrency behavior when twenty people open dashboards at 09:00.

Option C — hybrid

Lakehouse (Delta or Iceberg) for bronze and silver on object storage; warehouse for gold.

For: cheap raw retention with ACID guarantees; row-level deletes; time travel; and analysts still get a warehouse.

Against: two systems. Two access models. Two things to monitor. Compaction and vacuum as standing duties. For a four-person team, this is a real cost and the strongest argument against.

The Analysis

The team spent two weeks on it. Most of that time went into arguments that turned out not to matter, which is normal and worth documenting.

What they argued about first, and why it did not decide anything

Query performance. Benchmarked all three on the ten most expensive existing queries. The warehouse won, the lakehouse was within roughly 20%, the plain lake was highly variable depending on file layout.

This did not decide anything, because 20% on queries that already complete in seconds is not worth an architecture. They had been arguing about it for three days.

Storage cost. Using the frozen basis, two years of bronze clickstream (341 GB/year as Parquet):

$$682 \text{ GB} \times \$0.023 = \$15.69 \text{ per month}$$

Against warehouse storage at roughly the same volume, perhaps three to five times that. A difference of tens of dollars a month.

💸 Cost Check — When the cost difference is too small to decide anything

Storage cost difference between the options: on the order of $50/month, or $600/year.

Team cost: four engineers. Even one engineer-week of additional annual operational burden from running two systems substantially exceeds $600.

The cost analysis therefore argued against the hybrid — the option the team eventually chose. That is worth stating plainly, because cost arithmetic is a tool for finding which considerations are decisive, and here it demonstrated that cost was not one of them.

The lesson generalizes and it cuts against this book's own emphasis: run the numbers, and then be willing to conclude that the numbers do not decide it. A team that had let the cost analysis drive would have chosen warehouse-only and hit the privacy problem below eight months later.

The two considerations that actually decided it

1. Row-level deletion for privacy requests.

The legal question. Under a GDPR erasure request, Kestrel must be able to delete an individual's personal data across all systems, within one month of the request (extendable to three for complex cases). The clickstream contains IP addresses, device identifiers, and — after login — a customer identifier. It is unambiguously personal data.

What deletion costs under each option:

This eliminated Option B decisively. Not on cost, not on performance, but on a legal obligation with a statutory clock that the architecture had to be able to meet as a matter of routine rather than as a project.

🔐 Privacy & Governance — The requirement that arrives after the architecture

Notice when the privacy requirement showed up: after the team had spent three days on query benchmarks. It was not in the original problem statement. It arrived from the legal team, phrased as a question rather than a requirement, and it turned out to be the most architecturally consequential input in the entire decision.

This is the normal case. Privacy, security, and compliance requirements almost always arrive after engineering has framed the problem, and they frequently invalidate the framing. The practical response is not to become a lawyer. It is to ask three questions at the start of any storage decision:

1. Does any of this data identify a person, directly or in combination?
2. If someone asks us to delete their data, what is the procedure and how long does it take?
3. Is there a retention limit we are obliged to honor, and what enforces it?

Three questions, ten minutes, asked before the benchmarks. Chapter 31 covers the mechanics. This book is not legal advice; confirm your obligations with counsel.

2. Schema instability at the clickstream source.

The front-end teams ship weekly and add event properties without notice. Under Option A, every new property is either dropped at ingest or requires a schema migration in the warehouse — and dropping it means the data is gone permanently, because the source does not retain events.

Under Option C, bronze accepts the raw event as written; silver has an enforced schema; new properties sit in bronze until someone decides they matter. The raw layer absorbs the instability so the modeled layer does not have to.

This is the ELT argument from §3.3 in its most concrete form, and it is worth noticing that it is not really an argument about ETL versus ELT. It is an argument about who bears the cost of an unstable producer, and the answer is: a cheap, permissive raw layer, rather than a schema migration in the system your analysts query.

The remaining question: Delta or Iceberg

With Option C chosen, one decision remained, and it was genuinely close.

They chose Delta, on team experience and local tooling, and — this is the part worth copying — recorded it as the decision most likely to be revisited, with an explicit reversal condition and an estimated migration cost of three to four weeks.

The estimate is only three to four weeks because the underlying data is Parquet either way. The table format is a metadata layer. That is exactly the reversibility-buying move from §3.1: an open storage format converted a potentially expensive decision into a cheap one, before anyone needed it to.

The Decision

Option C, with Delta. The ADR is reproduced in full in §3.7 of the chapter.

Four things the team did that are worth stealing:

Wrote down the rejected options with reasons. A successor reading it in 2029 can tell that warehouse-only was considered seriously and why it lost, rather than assuming nobody thought of it.

Wrote reversal conditions as observable facts. Not "if requirements change" but "if total analytical data stays under ~2 TB for two years" — a condition you can write a query for.

Named the decision most likely to be wrong. Delta versus Iceberg, flagged in the document itself.

Set a review date. 2026-01-14.

What Happened

The team revisited the ADR on schedule in January 2026. Against the three reversal conditions:

"If total analytical data stays under ~2 TB for two years, collapse to warehouse-only." Not met. Bronze alone had reached 1.4 TB, and the trajectory put total analytical data past 2 TB within the year. Condition dead; the hybrid was earning its complexity.

"If Iceberg support in our engines clearly surpasses Delta, migrate." Genuinely arguable. The team judged it not clearly surpassed and deferred with a note. This is what a reversal condition is supposed to do — it does not decide for you, it makes you look on a schedule instead of never.

"If we adopt a second processing engine that reads Iceberg better, revisit immediately." Not met. No second engine adopted.

The unexpected finding came from a section the ADR did not have: the consequences the team had predicted were only about half right.

The knowledge-concentration effect is the most interesting one, because it is an organizational consequence of a technical decision, and it was invisible in every framework the team used to make the decision.

Lessons

1. Run the cost arithmetic, and be willing to conclude it does not decide anything. Here it argued mildly against the chosen option, and it was right to run it — that is how you find out which considerations are decisive.

2. Query benchmarks decided nothing. Three days on a 20% difference in queries that complete in seconds. Benchmark late, and only when a decision genuinely turns on performance.

3. Ask the privacy questions before the benchmarks. Three questions, ten minutes, and one of them eliminated an entire option.

4. A cheap permissive raw layer absorbs an unstable producer. That is the concrete form of the ELT argument, and it is really about who bears the cost of upstream instability.

5. Open formats buy reversibility in advance. Delta versus Iceberg is a three-to-four-week decision only because the bytes are Parquet either way.

6. Name the decision most likely to be wrong, in the document. It focuses the review and it makes the eventual reversal a scheduled event rather than an embarrassment.

7. Reversal conditions must be observable. "If requirements change" is not a condition. "If bronze exceeds 2 TB" is.

8. Technical decisions have organizational consequences that no technical framework surfaces. The knowledge concentration among two of four engineers was a real cost of the hybrid and nobody saw it coming.

Questions for Discussion

1. The cost analysis argued against the chosen option. Should the team have weighted it more heavily? Construct the scenario in which choosing warehouse-only would have been correct, and say what Kestrel would have had to do about the deletion requirement.

2. Three days went into query benchmarks that decided nothing. What would have to be true for benchmarking to be the right first step? Is there a rule of thumb for when to benchmark early?

3. The privacy requirement arrived from the legal team as a question, after engineering had framed the problem. Design a process that surfaces it earlier without adding a compliance review to every technical decision.

4. Option B (plain lake) was eliminated on deletion cost. Is there a design that keeps a plain lake and makes deletion tractable — for example, storing personal identifiers in a separate keyed store and referencing them by token? What does that cost, and what does it break?

5. The Delta/Iceberg choice was made on team experience. Is that a good reason? Argue both sides, and say how you would weight "we know this one" against "this one is technically better."

6. Only two of four engineers ever did lakehouse work. Whose responsibility is that, and what would you have done at the six-month mark to prevent it?

7. The ADR's review found one condition dead, one arguable, one unmet. Is an annual review the right cadence? What would make you review sooner, and what would make an ADR not worth reviewing at all?