Key Takeaways: The Medallion Architecture

DataField.Dev

Key Takeaways: The Medallion Architecture

The one thing

Everybody draws three boxes; the boxes are free. The value of this pattern is entirely in the guarantees at each boundary and in a build that fails when one is violated — and almost nobody writes the guarantees down.

What the layers are for

🔎 They convert an unbounded search into a binary search. A number is wrong; where do you look?

wrong in GOLD, right in SILVER    -> a business rule. One model, usually one line.
wrong in SILVER, right in BRONZE  -> a type, a dedup, or a key. Still narrow.
wrong in BRONZE                   -> not your bug — and you can PROVE it.

The third outcome pays for the pattern. "The source sent us this" is defensible only if you kept what the source sent. Without a faithful bronze, every upstream data problem is an argument you cannot win.

They do not improve data quality — assertions do. They provide somewhere to put a check where it means something, and somewhere to put data that fails one.

The three guarantees

Bronze: exactly what the source sent, in arrival order, nothing removed, nothing interpreted. Append-only. Untyped. Keeps what you do not want — filtering at ingestion destroys the evidence.

📐 Plus four envelope columns: _ingested_at, _source_file, _batch_id, _row_number. The test for whether a column belongs: could the source have sent it? If yes it is data and must be faithful; if no it is envelope and it belongs.

Silver: correct types, one row per entity per version, consistent names and units — and no decision anyone could disagree with.

⚠️ The test: if two competent people could disagree, it is a business rule. Nobody disagrees that "142" is 142; people do disagree about what -1 means, whether "" is null, and what timezone an offsetless timestamp is in. When unsure, push it up — a business rule in silver is invisible to its owner; a conformance rule in gold is merely repetitive.

Gold: a business question, at a documented grain, with named and owned definitions. And gold is disposable — a gold table holding the only copy of something is a source, not gold.

Enforcement

Nine rules a build can run: backflow · layer-skip · mutable-bronze · bronze-interprets · silver-no-key · business-logic-in-silver · cast-in-gold · gold-no-grain · unreconstructable.

Kestrel: 8 violations across 6 of 17 models, in a graph that looked fine after two years.

🏭 One defect can trip two rules, and that is the design working. A gold model reading bronze must therefore cast. Fix the blocking finding and the warning disappears — suppressing the warnings instead leaves the boundary violation in place and removes the second signal.

📐 Classify a rule by asking: if this is violated, is any number wrong? Yes → blocking. No → warning. It has kept the blocking set small enough that nobody has asked to turn it off.

Tests belong at different layers

📐 Bronze asserts arrival, never correctness. A bronze table full of garbage is doing its job; an assertion that quantity > 0 fires on every return and teaches the team that assertions are negotiable.

Silver asserts types, keys, and integrity. Gold reconciles to source — and only gold does.

Do not repeat a silver assertion in gold. Kestrel found 31 duplicated assertions, all added by people who did not know the silver test existed.

The exception is a row count, which belongs at both: bronze's detects a truncated source, silver's detects a bad transformation. Two assertions are duplicates if they fail for the same reason, not if they compute the same number.

What the layers buy

corrupt model              models to rebuild      cost
bronze.customers_raw                       9   $102.00
silver.stg_orders                          5    $25.92
gold.daily_revenue                         1     $0.96
                     full rebuild                $198.96

💸 The entire platform rebuilds for $198.96 — less than one erasure request from unpartitioned bronze Parquet. So rebuild quarterly, not to fix anything but to confirm you still can.

Kestrel's has failed three times in two years: a vendor that corrected history in place, a gold model depending on a hand-made table no lineage tool knew about, and a non-deterministic dedup. None is detectable by any assertion, because each is a property of the whole graph over time.

And it is only cheap because bronze exists. The bronze layer costs $333.11/month to store — the price of the option, and the best-value line item in the book.

Depth and blast radius

📏 bronze.customers_raw reaches 47% of the graph. gold.daily_revenue reaches nothing.

Test breadth by blast radius, test depth by visibility. Effort is normally allocated by how interesting a model is, which is close to backwards.

🔎 A silver model deeper than the gold it reads is impossible in a correct graph. Depth 3 against depth 2, where every other silver model is depth 1.

Depth tests the graph's shape rather than its labels, so it survives mislabelling, moving, and renaming — and it is more persuasive, because a rule violation is arguable and a stg_ model at depth 3 is a self-evident contradiction.

Quarantine

⚠️ Do not fail the load; do not drop and log. Quarantine. With four properties:

A size limit that pages — 0.1% of a load or 500 rows.
The assertion that rejected it, not just "failed."
Replayable — _batch_id plus the full source row.
A retention that forces the drain. This is the one that fails.

Kestrel's reached 214,000 rows and an analyst queried it into a monthly report, wrong by 1.8% — because pre-conformance rows have source types, source names, and no deduplication. A queue with a retention is a queue somebody empties; a queue without one is a table.

Drift

⚠️ Drift is the normal end state, by the same ratchet as Chapter 25's alerts, Chapter 30's grants, and Chapter 33's idle warehouses:

the shortcut:    four lines, a requester, a deadline
doing it right:  a new model, a grain, an owner, a test
what revisits working code:  nothing

The fix is not discipline — the shortcut must fail the build, with an expiring suppression as the escape hatch. Nine of eleven suppressions were resolved, seven in the week their expiry approached.

And a violation that persists is a template. Two of Kestrel's later violations were created by engineers who checked what the codebase did rather than what the doc said — which is the correct thing for a new engineer to do. The cost of an unfixed violation is its own cost plus every copy of it.

Costs, honestly

Storage — bronze 13.15 TiB, silver and gold 0.83 TiB; 2.1% of the bill. The cheapest.

Compute — every row written two or three times. The real cost, and it is in the expensive category.

Latency — about 40 minutes of Kestrel's 3.5-hour path is layer overhead.

And the arguments. §34.3's boundary cases each took a real discussion. Not waste — the discussion is where a business rule gets an owner — but it is senior engineering time and a team adopting this should expect it.

When not to

One source, one consumer, no conformance problem → silver is a pass-through; use two layers.

A long-retention log that can be replayed → the log is bronze.

When you will not enforce it. The honest one. Three directories with no rules give you all three costs and none of the benefit. One well-tested transformation layer beats three untrusted ones.

And never add a layer because a different team owns it — that is a mesh domain in a medallion costume, and Chapter 35 is where it belongs.

The code

code/layer_check.py — nine rules over a model graph, a replay planner with costs, depth and blast-radius reporting, and a drift check comparing declared layer against actual behaviour. Thirty-nine self-checks. --demo runs all four; Exercise 34.11 adds three rules of your own.