Chapter 34: The Medallion Architecture

"We had a bronze folder, a silver folder, and a gold folder. What we did not have was any statement of what was true about a table in one of them rather than another, which meant we had three folders."

Overview

Everybody draws three boxes. The boxes are free.

The medallion architecture — bronze, silver, gold — is the best-known pattern in data engineering and the one most often adopted as decoration. Creating three directories takes a minute and changes nothing. What changes something is a set of guarantees at each boundary, and a build that fails when one is violated.

This chapter is about the guarantees, and its central claim is unpopular in both directions:

The pattern is worth more than its reputation among skeptics, because it buys one specific and enormously valuable property — when something is wrong, it tells you where to look, and it makes "rebuild from raw" an option rather than a wish.

And it is worth far less than its reputation among adopters, because none of that value comes from the folder names. It comes from the rules, and almost nobody writes the rules down.

Kestrel has used these layer names since Part II. Chapter 5 §5.6 put four buckets in the object store; Chapter 9 partitioned bronze; Chapter 19 built silver and gold as dbt models; Chapter 20 made gold incremental. This chapter formalizes what has been assumed for twenty-nine chapters, and then code/layer_check.py enforces it — finding eight violations across six models in a seventeen-model graph that looked fine.

34.1 What the Layers Are Actually For

Three claims are usually made for the medallion architecture. Two of them are weak.

"It separates concerns." True and vague. Every architecture separates concerns; the question is which concerns and at which boundary.

"It improves data quality." False as stated. Layers do not improve quality — Chapter 23's assertions do. What layers provide is somewhere to put a quality check where it means something, and somewhere to put data that fails one.

"It makes debugging tractable." This is the real one, and it is worth more than the other two combined.

🔎 Read the Plan — the question the layers answer

A number is wrong on a dashboard. Where do you look?

Without layers, the honest answer is "anywhere between the source system and here." The investigation is unbounded, and its length is proportional to the size of the transformation codebase.

With enforced layers, it is three questions, in order, and each is cheap:

```text 1. Is the number wrong in GOLD but right in SILVER? -> a business rule is wrong. One model. Usually one line.

2. Is it wrong in SILVER but right in BRONZE? -> a type, a dedup, or a join key. Still narrow.

3. Is it wrong in BRONZE? -> it is not your bug. It arrived that way, and you can PROVE it, because bronze is what the source sent. ```

The third outcome is the one that pays for the pattern. "The source sent us this" is a defensible statement only if you kept what the source sent, unmodified, and can produce it. Without a faithful bronze, every upstream data problem becomes an argument you cannot win, and Kestrel spent two years losing those arguments before Part II.

The general form: the layers convert an unbounded search into a binary search. That is the whole value proposition, it is a debugging property rather than a purity property, and it only holds if each layer actually guarantees what it claims — which is §34.5.

34.2 Bronze: What the Source Sent

Bronze is a record of what arrived, and nothing else.

The guarantee:

A bronze table contains exactly what the source system sent, in arrival order, with nothing removed and nothing interpreted — plus metadata about the arrival itself.

Four consequences, and the fourth is the one teams get wrong:

It is append-only. Never updated, never deleted from, never rebuilt in place. A bronze table that is rebuilt nightly is not bronze, and every replay guarantee in §34.9 depends on this.

It is untyped, or minimally typed. A sku that arrives as "00471" is stored as "00471", not as 471. Chapter 22 Case Study 2's leading zeros are exactly this — the type decision belongs to silver, where it is visible and revisable.

It keeps what you do not want. Malformed rows, unknown columns, duplicates. Filtering at ingestion destroys the evidence that lets you answer question 3 above.

It carries arrival metadata, which is not a violation of fidelity but a requirement:

_ingested_at    TIMESTAMP     -- when we received it
_source_file    TEXT          -- which file or partition
_batch_id       TEXT          -- which run
_row_number     BIGINT        -- position within the source

📐 Design Decision — the four columns are not a compromise on fidelity

A purist objection: adding columns to bronze means bronze is not what the source sent.

The distinction that resolves it: these columns describe the arrival, not the record. They are the envelope, not the letter. Nothing in them changes, filters, reinterprets, or hides a source value, and removing any source value would be a fidelity violation while adding _ingested_at is not.

And each of the four earns its place by answering a question you will be asked:

• _ingested_at — "when did we get this?" Different from any business timestamp, and the only one you control. Chapter 13's four ways updated_at lies.
• _source_file — "which file was this in?" The unit of re-ingestion when a file is bad.
• _batch_id — "which run produced these rows?" The unit of deletion when a run must be undone, and therefore the thing that makes Chapter 20's idempotency achievable at all.
• _row_number — "what order did they arrive in?" Needed for any deduplication that has to break a tie deterministically (Chapter 20's cdc_lsn problem).

The test for whether a column belongs in bronze: could the source system have sent it? If yes, it is data and it must be faithful. If no — if it is a fact about your pipeline — it is envelope, and it belongs.

34.3 Silver: Typed, Deduplicated, Conformed

Silver is bronze made usable, with no opinions added.

The guarantee:

A silver table has correct types, one row per entity per version, consistent names and units across sources — and encodes no business decision that anyone could disagree with.

The last clause is the whole difficulty.

What silver does:

What silver does not do: decide that a customer is active, that revenue is net of refunds, that a session ends after 30 minutes, or that an order is fraudulent. Every one of those is a decision someone owns (Chapter 30 §30.8), and putting it in silver hides it from the people who own it.

⚠️ Failure Mode — the cast that is secretly a business rule

The boundary is less obvious than the table above suggests, and three cases at Kestrel took a real argument:

A null country code. Silver may not invent "US". But is coercing "" to NULL a cast or a rule? Kestrel's answer: a cast, because the source's empty string means absent — and the reasoning was written down, which is the actual requirement.

A negative quantity. The source occasionally sends quantity = -1 for a return. Is turning that into a separate is_return flag a conformance or a business rule? Kestrel's answer: a business rule, because deciding that -1 means "return" is an interpretation of the source's convention — even though it is obviously correct. It went to gold.

A timestamp with no timezone. The source sends 2026-03-14 09:00:00 with no offset. Assuming America/New_York is a business decision that looks exactly like a parse, and it is the one that caused a real incident: two silver models assumed different timezones for the same column, for eleven months.

The test that resolved all three: "if two competent people could disagree about the answer, it is a business rule." Nobody disagrees that "142" is 142. People can and do disagree about what a -1 means, and the moment they can, the decision needs an owner, a name, and a place in gold where it is visible.

The cost of getting this wrong is asymmetric. A business rule in silver is invisible to the people who own it and gets rediscovered during an audit. A conformance rule in gold is merely repetitive — you write the same cast in four models. So when genuinely unsure, push it up.

34.4 Gold: Business Definitions, Consumed

Gold is what people query, and every row of it encodes decisions somebody owns.

The guarantee:

A gold table answers a business question, at a documented grain, using definitions that are named, owned, and defined once.

Three requirements, all of them from Chapter 30:

A documented grain. "One row per order line, per day, after refunds." §30.2's field, and the one that prevents the most errors.

Named definitions. is_active_customer is computed in one place (§30.8's certified model), not re-derived per mart.

An owner who can act. §30.4.

And one property that is this chapter's: gold is disposable. Every gold table must be reconstructible from silver, which is reconstructible from bronze. A gold table containing the only copy of something is not gold — it is a source, and it belongs in bronze with a note about where it came from.

34.5 Nine Rules a Build Can Enforce

Here is the difference between the pattern and its decoration. code/layer_check.py implements nine rules over a model graph and runs in CI:

R1  backflow                  a model must never read a HIGHER layer
R2  layer-skip                gold must not read bronze directly
R3  mutable-bronze            bronze must be append-only
R4  bronze-interprets         bronze must not cast or apply rules      (warn)
R5  silver-no-key             silver must declare a unique key
R6  business-logic-in-silver  silver must not encode business rules
R7  cast-in-gold              gold must not cast                       (warn)
R8  gold-no-grain             gold must document its grain             (warn)
R9  unreconstructable         gold must reach bronze

Run against Kestrel's seventeen-model graph:

severity  model                        rule                     detail
────────────────────────────────────────────────────────────────────────
blocking  bronze.supplier_feed         mutable-bronze           materialized=table
blocking  gold.fct_session             layer-skip               reads bronze.clickstream_raw
blocking  silver.stg_customer_flags    business-logic-in-silver performs business_rule
blocking  silver.stg_order_enriched    backflow                 reads gold.dim_customer
blocking  silver.stg_suppliers         silver-no-key            nothing deduplicates
warning   bronze.supplier_feed         bronze-interprets        performs cast
warning   gold.customer_summary        gold-no-grain            no documented grain
warning   gold.fct_session             cast-in-gold             silver left the column untyped
────────────────────────────────────────────────────────────────────────
  5 blocking, 3 warning — 8 findings across 6 models

Two things in that output are worth more than the list.

🏭 From the Pipeline — one defect, two rules, and why that is the design working

gold.fct_session trips two rules, and they are not two problems. They are one problem seen from two directions.

It reads bronze.clickstream_raw directly (R2, layer-skip). Because bronze is untyped, it must therefore cast (R7, cast-in-gold). The second finding is a mechanical consequence of the first, and this is exactly what you want from a rule set: a boundary violation should be visible from more than one angle, because you may only be looking from one of them.

The same coupling holds for bronze.supplier_feed, which is mutable (R3) and casts (R4) — a table being rebuilt in place is a table someone is transforming, and the two findings arrive together.

The practical consequence for whoever fixes it: do not fix the warnings. Casting in gold and interpreting in bronze are symptoms; the blocking findings are the disease. Kestrel's first attempt at this suppressed the two warnings — added the casts to a permitted list — which left both boundary violations in place and removed the second signal.

When one defect produces findings at two severities, fix the severe one and the other disappears. If it does not disappear, you had two problems, which is also useful to know.

And the severity split is deliberate.

📐 Design Decision — why a missing grain is a warning and backflow is a blocker

Six deliberate defects, eight findings, and only five blocking. The distinction is not importance.

A blocking rule breaks a property the graph depends on. Backflow makes the graph uncomputable from raw. Mutable bronze destroys replay. A silver model with no key does not deduplicate, so every downstream count is wrong. These are not stylistic.

A warning harms a reader. A missing grain (§30.2) will cause somebody to write a wrong join, and that is a real cost — but the graph still computes, the numbers are still right, and blocking a deploy on it teaches the team to disable the check.

The rule Kestrel uses to classify a new check: "if this is violated, is any number wrong?" Yes → blocking. No → warning. It is crude, it is defensible in a code review, and it has kept the blocking set small enough that nobody has asked to turn it off — which is the only measure of a lint rule that matters (Chapter 27 §27.11).

34.6 Tests Belong at Different Layers

Chapter 23 gave you twenty-two assertions and no guidance on where to put them. The layers answer that, and the answer is not "everywhere."

A test at the wrong layer is either a false alarm or a missed defect, and which one depends on the direction of the mistake.

Three rules that fall out, and each one removes a class of wasted work.

Do not assert correctness in bronze. Bronze's job is to be faithful, which means a bronze table containing garbage is a bronze table doing its job. An assertion that quantity > 0 in bronze fires every time the source sends a return, and the correct response is always to weaken the assertion — which teaches the team that assertions are negotiable.

Do not repeat a silver assertion in gold. If order_id is unique in silver, it is unique in gold, and the duplicate test costs compute forever while telling you nothing new. Kestrel found 31 such duplicated assertions, all added by people who did not know the silver test existed.

Do reconcile at gold, and only at gold. "Does daily_revenue sum to what the source system says?" is the most valuable assertion in the platform (Chapter 23 §23.11) and it is meaningless anywhere else, because the intermediate layers are deliberately not the business's numbers.

📐 Design Decision — the assertion that belongs at two layers, and why it is not a duplicate

Row counts are the exception, and understanding why sharpens the rule.

Bronze asserts: "today's file has between 5,000 and 9,000 order rows." This is a statement about the source — it detects a truncated export, a partial upload, a source outage.

Silver asserts: "stg_orders gained between 5,000 and 9,000 rows, and the count equals bronze's count minus duplicates." This is a statement about the transformation — it detects a dedup that removed too much, a join that fanned out, a filter someone added.

They look identical and they fail for opposite reasons. When bronze's fires, the source is wrong and you have an upstream conversation. When silver's fires, you are wrong.

Kestrel had only the silver one for eighteen months, which meant every truncated source file presented as a pipeline bug, and the on-call engineer spent the first twenty minutes of each incident establishing that the platform was fine — twenty minutes that Chapter 26 §26.5's decision tree now saves by asking "is the bronze count within band?" first.

The general test: two assertions are duplicates if they fail for the same reason, not if they compute the same number.

34.7 Where the Boundaries Actually Fall

Six hard cases, and the reasoning matters more than the answers.

The last one has a subtlety worth stating. Excluding is gold — but the row stays in bronze and silver. A gold table that excludes something and a silver table that never had it are different systems: only the first one can tell you what was excluded and why.

34.8 How Many Layers?

Three is a convention, not a law, and both neighbours are defensible.

Two layers (raw + curated) is right when there is one source, one consumer, and no conformance problem. Silver exists to reconcile multiple sources; with one source it is a pass-through, and a pass-through layer is pure cost.

Four or five layers appear at scale, and the extra ones are almost always:

• A quarantine layer for rows that failed a Chapter 23 assertion — arguably part of silver, and worth separating when the volume justifies its own retention.
• A serving or presentation layer above gold: pre-aggregated, denormalized, shaped for one BI tool. This is real and the reason to separate it is that its correctness constraints are weaker.

What is never a good reason for another layer: an organizational boundary. A layer that exists because a different team owns it is a data mesh domain wearing a medallion costume, and Chapter 35 is where that belongs.

34.9 What the Layers Buy: Replay

Here is the payoff, priced. --replay takes a model you have discovered is wrong and reports everything that must be rebuilt, in dependency order, with the cost:

REPLAY -- silver.stg_orders is wrong. What must be rebuilt?

  model                           nodes   hours       cost
  ──────────────────────────────────────────────────────────
  silver.stg_orders                   4    0.50      $4.80
  gold.fct_order_line                 8    0.90     $17.28
  silver.stg_order_enriched           2    0.30      $1.44
  gold.customer_summary               2    0.30      $1.44
  gold.daily_revenue                  2    0.20      $0.96
  ──────────────────────────────────────────────────────────
  TOTAL (5 models)                         2.20     $25.92

And the cost depends entirely on how far down the corruption is:

corrupt model              models to rebuild    hours      cost
bronze.customers_raw                       9     4.00   $102.00
bronze.orders_raw                          6     2.40    $26.88
silver.stg_orders                          5     2.20    $25.92
gold.fct_order_line                        3     1.40    $19.68
gold.daily_revenue                         1     0.20     $0.96
                    full rebuild of everything          $198.96

💸 Cost Check — the whole platform rebuilds for $198.96

That number is the argument for this chapter and it is smaller than most people expect.

A complete reconstruction of every model from raw — seventeen models, every layer — costs $198.96 on Chapter 33's rate card. A single erasure request from unpartitioned bronze Parquet cost more than that (Chapter 31 §31.5), and Chapter 1's broken job cost 19× a full rebuild, every night.

Two conclusions follow, and the second is the useful one:

Rebuilding is cheap, so treat it as routine. Kestrel rebuilds everything from bronze once a quarter — not to fix anything, but to confirm that it still can. The rebuild has failed three times in two years: twice because a source system's historical export had changed, once because a model had acquired a dependency on a manually-created table. All three were found by an exercise that costs $198.96 and four hours.

And rebuilding is only cheap because bronze exists. Without a faithful, retained bronze, the "rebuild" is a re-extraction from source systems that may have overwritten their own history — which is not a rebuild, is not always possible, and is never $198.96.

The storage that makes this possible is Chapter 33's 2.1% line. Kestrel's entire bronze layer costs $333.11 a month to store, against a platform bill of $30,511. That is the price of the option, and it is the best-value line item in the book.

34.10 Depth and Blast Radius

Two numbers per model, and neither is in anyone's dashboard.

Depth is distance from raw. Blast radius is how many models a corruption in this one invalidates.

model                          layer     depth  affects
bronze.customers_raw           bronze        0        8   <-- widest
silver.stg_customers           silver        1        6
bronze.orders_raw              bronze        0        5
gold.dim_customer              gold          2        5
gold.daily_revenue             gold          4        0

bronze.customers_raw reaches eight of seventeen models — 47% of the graph. It is a small table, loaded by a simple job, and it is the most dangerous object in the platform.

📏 Scale Note — blast radius should determine test coverage, and never does

Test effort is usually distributed by how interesting a model is. The complicated aggregation gets six assertions; the customer load gets a not-null check.

Blast radius says the opposite. bronze.customers_raw is trivially simple and can invalidate 47% of the graph; gold.daily_revenue is the most business-critical table in the platform and can invalidate nothing, because nothing reads it but a dashboard.

Kestrel now weights Chapter 23's register by blast radius, and the reallocation was substantial: three assertions moved off daily_revenue and onto the two bronze loads that feed half the graph.

The counter-argument, which is fair: a defect in daily_revenue is seen by the CEO at 06:15 and a defect in customers_raw is seen by nobody until it propagates. Both matter, and they are different risks — visibility versus reach. The resolution Kestrel settled on: test breadth by blast radius, test depth by visibility.

And one diagnostic falls out of the depth calculation for free.

🔎 Read the Plan — the silver model that is deeper than the gold it reads

text silver.stg_order_enriched silver depth 3 gold.dim_customer gold depth 2

A staging model further from raw than a mart is impossible in a correct medallion graph, and it is the clearest possible symptom of backflow.

Every other silver model in Kestrel's graph is depth 1. This one is depth 3 because it reads a gold table, which reads a silver table, which reads bronze.

Why this diagnostic is worth having when R1 already catches backflow: it survives renaming. A team that reorganizes directories, or adopts different layer names, or has a model whose layer is mislabelled, still cannot produce a staging model at depth 3 without something being wrong. The depth check tests the graph's shape rather than its labels, and §34.13 is about how quickly the labels stop being trustworthy.

34.11 What the Pattern Costs

Three costs, stated honestly, because most writing on this pattern states none.

Storage. The same data exists at least twice, sometimes three times. Kestrel: bronze 13.15 TiB, silver and gold together 0.83 TiB — and per Chapter 33 §33.4, this is 2.1% of the bill. The cheapest of the three by a wide margin.

Compute. Every row is written two or three times. This is the real cost and it is a compute-layer cost, which is where the money is.

Latency. Each layer adds a hop. Kestrel's bronze-to-gold path is about 3.5 hours end to end, of which perhaps 40 minutes is layer overhead rather than work.

And one cost nobody lists: the arguments. §34.3's three boundary cases each took a real discussion. That is not waste — the discussion is where the business rule gets an owner — but it is a cost, it is paid in senior engineering time, and a team adopting this pattern should expect it.

34.12 Quarantine: Where Bad Rows Live

A row fails a silver assertion. What happens to it?

Three answers, and only the third scales.

Fail the whole load. Correct, safe, and it means one malformed row from a supplier stops the 6am report. Right for a correctness-critical pipeline and wrong for most others.

Drop it and log it. The load succeeds, the row is gone, and the log line is read by nobody. This is what most pipelines do and it is silent data loss — Chapter 23's whole complaint.

Quarantine it. The row goes to a parallel table with the reason it failed, the load proceeds with the rest, and the quarantine is a monitored queue with an owner.

-- silver/_quarantine/stg_orders_rejected
order_id, <all source columns>,
_rejected_at    TIMESTAMP,
_rejected_by    TEXT,      -- which assertion
_rejected_value TEXT,      -- what the offending value was
_batch_id       TEXT       -- so it can be replayed

Four properties that make a quarantine work rather than become a second landfill:

It has a size limit that pages. A quarantine growing without bound is a dropped-row log with extra steps. Kestrel pages when it exceeds 0.1% of a load or 500 rows, whichever is smaller.

It records the assertion, not just the failure. "Failed" is useless; "failed order_id not null" is a fix.

It is replayable. _batch_id and the full source row mean a corrected row can be re-admitted without re-ingesting the source — which is Chapter 24 §24.14's schedulable replay job.

It is emptied. This is the one that fails. A quarantine nobody drains is a quarantine that grows until somebody truncates it, and the rows in it are lost with no record.

⚠️ Failure Mode — the quarantine that became a second source of truth

Kestrel's quarantine held 214,000 rows at its peak, accumulated over fourteen months, and the problem was not the volume.

An analyst discovered it and started querying it. Their reasoning was entirely sound: "these are real orders that the platform rejected; my revenue figure should include them." They were right that the orders were real. They were wrong that the rows were usable — the quarantine holds pre-conformance rows, with source types, source column names, and no deduplication, and the analyst's query double-counted every row that had been rejected twice.

The number reached a monthly report before anyone noticed, and it was wrong by 1.8%.

Three responses, and the third is the one that generalizes:

• The quarantine is internal in the catalog (§30.2) and does not appear in search.
• Access is restricted to the platform team.
• The quarantine now has a retention of 30 days, which forces the drain. A queue with a retention is a queue somebody empties; a queue without one is a table.

The deeper lesson is about what a holding area attracts. Anything that persists and contains business data will eventually be queried by somebody who does not know why it exists — which is Chapter 30 §30.2's internal tier and Chapter 31's classification arriving in a place nobody thought to apply them.

34.13 Layer Drift

The directory is not the guarantee. --drift compares each model's declared layer against what its SQL actually does:

bronze.supplier_feed        declared=bronze  behaves like=silver   cast,land
silver.stg_customer_flags   declared=silver  behaves like=gold     business_rule,cast,dedup

Two of seventeen models are in the wrong layer, and neither was moved there deliberately. Each acquired an operation over time — a cast added to a bronze loader to fix a downstream error, a business flag added to a staging model because that was where the data already was.

⚠️ Failure Mode — drift is the normal end state, and the mechanism is always the same

Nobody decides to put a business rule in silver. The sequence is:

1. Someone needs is_active_customer for a dashboard, urgently.
2. The data is already in stg_customers, joined and typed.
3. Adding one CASE WHEN there is four lines; doing it properly is a new gold model, a grain statement, an owner, and a test.
4. The four-line version ships, correctly, and works.
5. Nothing ever revisits it, because it is not broken.

Every step is locally reasonable and the outcome is a layer that no longer means anything — which is the same ratchet as Chapter 25's alerts, Chapter 30's grants, and Chapter 33's idle warehouses. This book has now found it in four places, and the shape is identical each time: the shortcut has a requester and a deadline; the correction has neither.

The fix is not discipline. It is that step 4 must fail the build. layer_check.py in CI turns a four-line shortcut into a four-line shortcut plus a red pipeline, at which point doing it properly is genuinely the faster path — which is the only version of this that holds.

And when the shortcut is genuinely necessary, the escape hatch is an explicit, expiring suppression:

```yaml

models/silver/stg_customer_flags.yml

meta: layer_check_ignore: rule: business-logic-in-silver reason: "needed for the Q3 board deck; move to gold by 2026-10-15" expires: "2026-10-15" ```

The expiry date is the mechanism — Kestrel's CI fails an expired suppression, which converts "we'll fix it later" into a dated commitment. Eleven suppressions have been created in two years and nine have been resolved, seven of them in the week their expiry approached.

34.14 When Not to Use It

Three conditions, and the first is common.

One source, one consumer, no conformance problem. Silver is a pass-through. Use two layers.

A pure streaming platform where "raw" is the log. If Kafka retains everything with a long retention and you can replay from it, the log is bronze and materializing a second copy may be redundant — though check Chapter 29 §29.7 on retention windows before relying on this.

When you will not enforce it. This is the honest one. Three directories with no rules give you the storage cost, the compute cost, and the latency cost of the pattern, and none of the debugging benefit, because you cannot trust that a silver table is deduplicated or that a bronze table is faithful.

If you are not going to write §34.5's rules, do not adopt the layers. A single well-tested transformation layer is better than three untrusted ones.

🎓 Interview Angle — "explain the medallion architecture"

A recall question with a recall answer — bronze raw, silver cleaned, gold business — and every candidate gives it.

The strong answer states what each layer guarantees to the next:

"Each layer's guarantee is what the next layer is allowed to assume. Bronze guarantees fidelity — this is what the producer sent, including the parts we don't understand yet — so it's append-only and we never fix anything in it. Silver guarantees shape: typed, deduplicated, conforming to the contract, and that's where a bad row gets rejected. Gold guarantees meaning: the business rules are applied and the numbers are the ones finance recognises. The reason bronze exists at all isn't tidiness — it's that you can rebuild everything downstream from it. Ours rebuilds for about $200 and four hours, and we run it quarterly, not to fix anything but to confirm we still can."

Four things that answer does. It frames layers as contracts between layers rather than as folders. It says what bronze is for, which almost nobody does. It gives the rebuild cost with a number. And the quarterly-rebuild-to-confirm line is the detail that lands, because it is obviously a thing somebody actually does.

The follow-ups:

"Where do you put a business rule?" — gold, and the interesting half is why not silver: a rule in silver means every consumer inherits a decision they cannot see, and silver is supposed to be the shape, not the meaning.

"What if bronze is expensive?" — price it. Kestrel's raw clickstream is 4.19 TB a year at $96 a month, and the thing it buys is the ability to re-derive silver when the parser was wrong (Exercise 16.23's 240×).

"How do you enforce the layers?" — a checker in CI, with rules that block and rules that warn, and suppressions that expire. A candidate who mentions the expiry has run one; an unexpiring suppression is a rule deleted with extra steps.

And the one worth volunteering: "three layers is a convention, not a law." Some platforms need a fourth for a presentation or semantic layer; some are fine with two. What is not optional is that each boundary has a stated guarantee, and a team that cannot state theirs has folders rather than layers.

🧱 Kestrel Platform — the nine rules, and what each one has actually caught

```text

rule severity caught, in 18 months

───────────────────────────────────────────────────────────────────────── 1 no business logic in a silver model BLOCK 4 -- a status re-mapping, twice 2 no gold model reads bronze directly BLOCK 2 3 bronze is append-only; no UPDATE BLOCK 1 -- a "quick fix" 4 every silver model has a grain test BLOCK 11 -- mostly new models 5 no model reads across two layers up BLOCK 3 6 no hardcoded schema names BLOCK 6 (ch 19 CS1) 7 every model has an owner warn 19 8 no SELECT * in a gold model warn 8 9 suppressions must have a reason and an expiry BLOCK 5 -- expired suppressions ```

Rule 9 is the one worth copying and the one people leave out. It makes the suppression mechanism self-cleaning: a suppression without a reason cannot be added, and one whose expiry has passed fails the build. Without it, the first inconvenient rule acquires a permanent exception and the checker becomes advisory.

Rule 4's eleven catches are almost all new models, which is the checker working as intended — it is a gate on the way in rather than an audit of what exists.

And rule 1 catching a status re-mapping twice is the finding that justifies the whole file. A CASE WHEN status IN (...) THEN 'complete' in a silver model is a business rule that every consumer inherits invisibly. Both instances were written by competent people who did not think of it as a rule, and neither would have been caught by review.

The two warnings are deliberately warnings. An unowned model should be visible and should not block a release at 5 p.m.; a warning that is reported weekly and trends downward is doing its job, and promoting it to a block would produce nineteen exceptions instead.

🧪 Try It — run the layer checker against a project you did not write

bash cd part-07-architecture-patterns/chapter-34-the-medallion-architecture/code python layer_check.py --self-check # 39 assertions python layer_check.py --check --project /path/to/your/dbt/project

Then read the findings in this order, which is not the order they print:

text 1. rule 6, hardcoded schema names -- a 30-second fix, and it is Chapter 19 CS1's defect 2. rule 1, business logic in silver -- the finding that matters most and looks least like a problem 3. rule 4, missing grain tests -- usually the largest count 4. rule 9, suppressions -- check whether any have expired; that is a rule about the rules 5. everything else

Rule 1 is the one to spend time on. A CASE WHEN status IN (...) in a silver model is a business rule that every downstream consumer inherits without seeing it. The checker flags the syntax; you have to decide whether it is a rule — and the decision is the exercise.

Two things to record:

The count, by rule, and how many are in models written in the last six months. A high proportion of recent models means the checker is needed as a gate; a high proportion of old ones means it is a backlog. Those need different responses and the split takes one git log per file.

And every suppression you add, with a reason and an expiry. Then set a calendar reminder for the earliest expiry. Rule 9 makes the mechanism self-cleaning and only if somebody looks at it once.

Expect the first run against a real project to produce dozens of findings. That is not a failure of the project; it is what an unenforced convention looks like when it is finally enforced, and the useful output is the ranked list rather than the total.

🔐 Privacy & Governance — bronze is where the obligation is heaviest and the tooling is thinnest

Bronze holds the most personal data, in the least structured form, for the longest time — which is the exact inverse of where governance tooling works well.

text layer personal data structured? retention erasure is ───────────────────────────────────────────────────────────────────── bronze ALL of it no -- an longest a rewrite of every opaque file containing them payload silver typed, tagged yes medium a partitioned delete gold aggregated or keyed yes shortest frequently nothing -- rebuild from silver

Three consequences follow from the top row and they are the price of Chapter 34's argument.

You cannot mask a column inside an opaque payload. Silver's column-level controls have no bronze equivalent; the granularity available is the file. That is the honest cost of the JSON-in-Parquet envelope (Chapter 11's 🔐) and it should be in the ADR.

The partitioning scheme decides the erasure cost, and bronze is partitioned by ingest date (§9.7) — which is the date least correlated with a person. An erasure sweep over bronze is therefore a full scan, and Kestrel's is priced at Chapter 31's figure for exactly this reason.

And bronze's retention is the longest by design. Two years of raw is two years of obligation, and it is the layer where "we kept it in case we need it" meets "why do you still have this."

What makes the trade defensible is that it is stated. ADR-003 records the retention and the reason; the deletion manifest includes bronze explicitly; the erasure cost is priced. A bronze layer nobody has thought about in these terms is the same architecture with none of the justification — and the difference is four sentences written when the layer is created.

One control that pays for itself: extract the identifier used for erasure into a real column at landing, as a deliberate, recorded exception to the raw-payload rule (Exercise 10.17's policy). It turns a full scan into a partition-pruned rewrite, and it is the single highest-leverage exception anyone makes to the bronze rule.

🧭 Version Note — "medallion" is a name, and the idea is older

The bronze/silver/gold vocabulary came from Databricks marketing around 2020. The idea is considerably older and appears under several names, which is worth knowing when you read anything written before then.

text name era the same three ideas ───────────────────────────────────────────────────────────────────── staging / ODS / data mart ~1995 land it, conform it, model it raw / refined / curated ~2015 same, in a Hadoop vocabulary bronze / silver / gold ~2020 same, with a table format making the write atomic

What is genuinely new in the third row is not the layering — it is that the boundaries are enforceable. A 1995 staging area was a schema you were asked not to query; a bronze layer with a table format, a CI checker, and an append-only guarantee is a boundary something can verify.

Two consequences for reading older material.

Advice about "the staging area" is usually advice about bronze and is usually still correct, with one exception: staging was temporary because storage was expensive, and bronze is retained, which is the change that makes Chapter 34's rebuild argument possible at all (Chapter 3 §3.3).

And advice about data marts maps onto gold almost exactly. Kimball's guidance on conformed dimensions and grain (Chapter 6) is thirty years old and unchanged, because it is about meaning rather than about cost.

The naming itself is worth being relaxed about. Some organisations use two layers, some four, some call them by domain names. What is not optional is that each boundary has a stated guarantee (§34.1) — and a team that has adopted the vocabulary without stating the guarantees has folders with fashionable names.

34.15 The Kestrel Platform

platform/layers/
  layer_check.py          # nine rules, in CI on every PR
  suppressions.yml        # explicit, expiring, with a reason
  replay.py               # "X is wrong" -> ordered rebuild plan + cost
  depth_report.py         # depth and blast radius, weekly

What changed after enforcement:

The line worth reading twice is the second-to-last. The quarterly rebuild has failed three times, each time revealing a dependency nobody knew existed — including a gold model that depended on a table an analyst had created by hand eighteen months earlier, which no lineage tool reported because the table was not in the project.

34.16 Summary

Everybody draws three boxes; the boxes are free. The value is entirely in the guarantees, and almost nobody writes them down.

🔎 The layers convert an unbounded search into a binary search. Wrong in gold but right in silver? A business rule. Wrong in silver but right in bronze? A type, a dedup, or a key. Wrong in bronze? It is not your bug — and you can prove it, which is a statement you can only make if you kept what the source sent.

Bronze is what the source sent: append-only, untyped, keeping what you do not want. 📐 Plus four envelope columns — _ingested_at, _source_file, _batch_id, _row_number — and the test for whether a column belongs is whether the source could have sent it.

Silver is typed, deduplicated, conformed — and encodes nothing anyone could disagree with. ⚠️ The test: if two competent people could disagree about the answer, it is a business rule. Nobody disagrees that "142" is 142; people do disagree about what -1 means. When unsure, push it up — a business rule in silver is invisible to its owner, while a conformance rule in gold is merely repetitive.

Gold answers a business question at a documented grain, with named and owned definitions — and it is disposable. A gold table holding the only copy of something is a source, not gold.

Nine rules a build can enforce, finding 8 violations across 6 models in a seventeen-model graph that looked fine.

🏭 One defect can trip two rules, and that is the design working. A gold model reading bronze must therefore cast. Fix the blocking finding and the warning disappears — Kestrel's first attempt suppressed the warnings and left both boundary violations in place.

📐 Classify a rule by asking: if this is violated, is any number wrong? Yes → blocking. No → warning. It has kept the blocking set small enough that nobody has asked to turn it off, which is the only measure of a lint rule that matters.

💸 The entire platform rebuilds from raw for $198.96. So do it quarterly, not to fix anything but to confirm you still can — Kestrel's has failed three times in two years, once on a gold model depending on a table an analyst made by hand. And it is only cheap because bronze exists; the bronze layer costs $333.11/month to store, which is the best-value line item in the book.

📏 Blast radius should determine test breadth and never does. bronze.customers_raw is trivially simple and reaches 47% of the graph; gold.daily_revenue is the most business-critical table and reaches nothing. Test breadth by blast radius, test depth by visibility.

🔎 A silver model deeper than the gold table it reads is impossible in a correct graph — depth 3 against depth 2, where every other silver model is depth 1. The depth check tests the graph's shape rather than its labels, which matters because the labels stop being trustworthy.

⚠️ Drift is the normal end state, by the same ratchet as Chapter 25's alerts, Chapter 30's grants, and Chapter 33's idle warehouses — the shortcut has a requester and a deadline; the correction has neither. The fix is that the shortcut must fail the build, with an expiring suppression as the escape hatch. Nine of eleven suppressions were resolved, seven in the week their expiry approached.

📐 Tests belong at different layers, and a test at the wrong one is a false alarm or a missed defect. Bronze asserts arrival, never correctness — a bronze table full of garbage is doing its job. Silver asserts types, keys, and integrity. Gold reconciles to source, and only gold does. Kestrel found 31 assertions duplicated from silver into gold. The exception is a row count, which belongs at both because the two fail for opposite reasons — and the test for a duplicate is whether two assertions fail for the same reason, not whether they compute the same number.

⚠️ Quarantine bad rows; do not drop them or fail the load. With a paging size limit, the assertion that rejected them, a replayable _batch_id, and a retention that forces the drain — Kestrel's reached 214,000 rows and an analyst queried it into a monthly report, wrong by 1.8%, because pre-conformance rows are not usable rows. A queue with a retention is a queue somebody empties; a queue without one is a table.

And do not adopt the layers if you will not enforce them. Three directories with no rules give you all three costs and none of the benefit. One well-tested transformation layer beats three untrusted ones.

Chapter 35 is data mesh — the pattern this book is most willing to say is usually implemented badly, and the one where §34.8's warning about a layer that exists because a different team owns it becomes the entire subject.

Key terms: bronze · silver · gold · source fidelity · append-only · progressive refinement · conformance · layer boundary · backflow · layer skipping · layer drift · blast radius · lineage depth · replay