Case Study 1: The Silver Model That Was Deeper Than Gold

"We had bronze, silver, and gold for two years. Then somebody wrote a script that measured how far each model was from raw data, and one of the staging models was further from raw than a mart."

Executive Summary

Kestrel ran the medallion architecture for two years with three directories and no rules. The first run of layer_check.py found eight violations across six of seventeen models.

The most consequential was invisible to every review: silver.stg_order_enriched read gold.dim_customer. A staging model depending on a mart — backflow — which meant the graph could not be computed from raw in dependency order, and a full rebuild would deadlock.

Nobody had noticed in eleven months. The DAG ran nightly and succeeded, because the scheduler happened to build dim_customer before stg_order_enriched and the cycle was never exercised.

The diagnostic that made it obvious was not the rule. It was a depth report:

silver.stg_order_enriched      silver     depth 3
gold.dim_customer              gold       depth 2
every other silver model       silver     depth 1

A staging model three hops from raw, reading a mart two hops from raw. Impossible in a correct medallion graph, visible in one column, and no directory listing shows it.

Skills applied: the nine layer rules (§34.5); depth and blast radius (§34.10); why a graph's shape is more trustworthy than its labels; and the quarterly rebuild that would have caught it in month one.

Background

The layers were adopted correctly and enforced never.

Kestrel's models/ directory had bronze/, silver/, and gold/ from the start. The team could articulate what each was for — the platform lead's onboarding doc said "bronze is what the source sent, silver is typed and deduplicated, gold is the business" — and by and large the models complied.

What did not exist was any mechanism. No rule, no test, no review checklist item. Compliance was a matter of everyone remembering, and everyone did remember, most of the time, for two years.

The eight violations that accumulated:

blocking  bronze.supplier_feed         mutable-bronze
blocking  gold.fct_session             layer-skip
blocking  silver.stg_customer_flags    business-logic-in-silver
blocking  silver.stg_order_enriched    backflow
blocking  silver.stg_suppliers         silver-no-key
warning   bronze.supplier_feed         bronze-interprets
warning   gold.customer_summary        gold-no-grain
warning   gold.fct_session             cast-in-gold

Six models of seventeen — 35% of the graph. Every one of them was created deliberately, by someone who knew the layer rules, under a deadline.

The Problem

The backflow is the one worth tracing, because its history is the most instructive.

Eleven months earlier, an analyst needed order data with the customer's current segment attached. The segment lives on gold.dim_customer, computed by an SCD Type 2 model.

The engineer had two options:

OPTION A (correct)
  create gold.fct_order_enriched, reading silver.stg_orders
  and gold.dim_customer. One new gold model, a grain statement,
  an owner, a test.                              ~half a day

OPTION B (what happened)
  add the join to silver.stg_order_enriched, which already
  existed and already read stg_orders.
                                                 ~20 minutes

Option B shipped. It was correct, it produced the right numbers, and it ran successfully every night for eleven months.

⚠️ Failure Mode — a cycle that never executes is still a cycle

Backflow does not fail. That is the entire problem.

The graph now contained an edge from silver up to gold. In dependency terms this is not a cycle — stg_order_enriched reads dim_customer and nothing reads stg_order_enriched back — so no topological sort fails and no scheduler complains.

What it destroys is the layer invariant, and the damage is entirely in what you can no longer say:

• "Silver can be rebuilt from bronze alone." No longer true. Rebuilding silver now requires gold, which requires silver.
• "To rebuild everything, do bronze, then silver, then gold." No longer true. The rebuild must interleave, and the correct order is now a property of the graph rather than of the layers.
• "A gold defect cannot corrupt silver." No longer true, and this is the dangerous one: a bad dim_customer now propagates downward, into a table whose name promises it is close to source.

None of these produce an error. They produce an architecture whose guarantees have quietly stopped being true, and the only way to find out is to try to use one of them — which Kestrel would have done at the first quarterly rebuild, and had never attempted one.

The general lesson: an invariant nobody exercises is an invariant nobody has. The layer promises are only real if something regularly depends on them, which is §34.9's argument for a rebuild you do not need.

The Analysis

The rule found it. The depth report explained it.

layer_check.py --check reported the backflow immediately, and the response from the team was initially defensive and reasonable: "it works, it's been fine for a year, and the alternative is a model that does almost the same thing."

What changed the conversation was one column of numbers.

🔎 Read the Plan — depth is more persuasive than a rule

text model layer depth affects bronze.customers_raw bronze 0 8 silver.stg_customers silver 1 6 silver.stg_orders silver 1 4 gold.dim_customer gold 2 5 silver.stg_order_enriched silver 3 0 <-- gold.fct_order_line gold 3 2 gold.daily_revenue gold 4 0

silver.stg_order_enriched sits at depth 3, alongside gold.fct_order_line. Every other silver model in the platform is depth 1.

This landed where the rule did not, and the reason is worth understanding. A rule violation is an assertion by a tool that something is against policy, and policy is arguable. A depth of 3 on a model named stg_ is a self-evident contradiction — the prefix says "close to source" and the number says "three transformations away."

And it is robust in a way the rule is not. The rule depends on the layer label being right. The depth calculation depends only on the graph's shape, so it survives a model being mislabelled, moved, or renamed, and it would still have flagged this model if someone had "fixed" the violation by relabelling it as gold.

A structural measurement beats a policy assertion, when you can get one. The same move appears in Chapter 30 §30.12 — reviewing against usage rather than against a list — and in Chapter 32 §32.7's row-level diff. Measure the property, not the declaration.

Then the second question: what would a rebuild have done?

attempted order:  bronze  ->  silver  ->  gold

  bronze.*                     ok
  silver.stg_orders            ok
  silver.stg_order_enriched    FAILS -- gold.dim_customer does not exist yet

The full rebuild — the property the entire pattern exists to provide — was broken, and had been for eleven months. Nothing had detected it because nobody had attempted one.

The Decision

Four changes, and the fourth is the one that prevents recurrence.

One: the backflow is removed. silver.stg_order_enriched becomes gold.fct_order_enriched, reading silver.stg_orders and gold.dim_customer. Half a day, eleven months later — the same half day Option A would have cost.

📐 Design Decision — the fix cost exactly what doing it right would have, and that is the argument

Option A was half a day. The fix was half a day. Nothing about the eleven-month delay made the correction more expensive.

Which sounds like an argument that the delay was free, and it is not, because the delay cost three things that do not appear in the half-day:

• Eleven months of a broken rebuild guarantee, unknown, unexercised, and load-bearing in the team's mental model of the platform.
• The migration of every downstream consumer. Two dashboards and a scheduled export referenced silver.stg_order_enriched by name, and renaming it required a deprecation window (§30.2's deprecated tier). That was three weeks of calendar time and about a day of work.
• The precedent. Two of the other five violations were created after this one, by engineers who had seen a silver model reading gold and reasonably concluded it was acceptable.

The third is the expensive one and it is the one nobody models. A violation that persists is not a static cost; it is a template. Kestrel's stg_customer_flags business-rule violation was created seven months after the backflow, by someone who checked what the codebase did rather than what the doc said — which is the correct thing for a new engineer to do.

The generalizable form: the cost of an unfixed violation is not its own cost. It is its own cost plus every copy of it. That is the argument for fixing the first instance quickly and it is much stronger than the argument for fixing any particular instance.

Two: layer_check.py runs in CI, blocking on the five blocking rules.

Three: expiring suppressions, per §34.13. Two of the eight were suppressed with dates rather than fixed immediately.

Four: a quarterly full rebuild from bronze. $198.96 and four hours.

What Happened

Nine days, of which the backflow was half a day of work and three weeks of calendar time.

The quarterly rebuild has since failed three times, and the three failures are the argument for the exercise:

A source system's historical export had changed. A vendor had corrected historical records in place, so replaying from their export produced different bronze than the bronze on disk. Not a bug — but a fact about the platform's reproducibility that nobody knew.

A model had acquired a dependency on a manually-created table. An analyst had built a mapping table by hand eighteen months earlier; a gold model read it; no lineage tool reported it, because the table was not in the dbt project. The rebuild failed on a missing relation, which is how it was found.

A silver model's dedup was not deterministic. Rebuilding produced a different surviving row for 41 customers, because the dedup broke ties on a column that was not unique. Chapter 20's cdc_lsn problem, found by a rebuild rather than by a test.

🏭 From the Pipeline — the rebuild is a test that nothing else can be

All three failures share a property: no assertion could have found them, because each is about a relationship between the platform and time rather than about the contents of a table.

• "Can we still reproduce what we produced?"
• "Is every input actually in the project?"
• "Is the output a function of the input, or of the order the input arrived in?"

These are properties of the whole graph, exercised only by running the whole graph from scratch. Chapter 23's register cannot express them; Chapter 27's CI does not run at that scale; Chapter 25's monitors watch a running system rather than a rebuildable one.

The exercise costs $198.96 and four hours, quarterly — call it $800 and two engineer-days a year. It has found three defects in two years that nothing else would have found, and one of them, the non-deterministic dedup, had been silently choosing between duplicate customer records for an unknown period.

The general principle: a capability you never exercise is a capability you do not have, and the cheapest way to find out is to schedule the exercise. Kestrel's rebuild is on the calendar, not in a runbook, for exactly the reason Chapter 26 §26.5 gives about runbooks nobody opens.

Lessons

1. Three directories and no rules is a naming convention. Kestrel complied "most of the time, for two years," which produced 8 violations across 35% of the graph.

2. ⚠️ Backflow does not fail. It is not a topological cycle, no scheduler complains, and it runs correctly every night — it just quietly makes three of the layers' promises untrue.

3. An invariant nobody exercises is an invariant nobody has. The full-rebuild guarantee had been broken for eleven months and nothing detected it because nobody had attempted one.

4. 🔎 Depth is more persuasive than a rule, because a rule violation is arguable and a stg_ model at depth 3 is a self-evident contradiction.

5. A structural measurement beats a policy assertion. Depth depends on the graph's shape, not on the layer label, so it survives mislabelling and renaming. Same move as reviewing access against usage (§30.12) and diffing features row-by-row (§32.7).

6. The 20-minute shortcut and the half-day fix were both still available eleven months later. The delay did not make the correction more expensive.

7. 📐 But the delay cost three things the half-day does not include: a broken guarantee nobody knew about, a three-week deprecation for two dashboards and an export, and the precedent.

8. The cost of an unfixed violation is its own cost plus every copy of it. Two later violations were created by engineers who checked what the codebase did rather than what the doc said — which is the correct thing for a new engineer to do.

9. 🏭 The quarterly rebuild found three defects nothing else could, because all three are properties of the whole graph over time rather than of any table's contents.

10. A capability you never exercise is a capability you do not have. Put the rebuild on the calendar, not in a runbook.

Questions for Discussion

1. The backflow ran correctly for eleven months. Was it actually a problem before the rebuild was attempted, or only once someone wanted the guarantee?

2. §"Design Decision" argues an unfixed violation is a template. How would you measure that effect, and would the measurement change anyone's priorities?

3. Depth persuaded where the rule did not. Construct another structural measurement that would catch a layer violation the nine rules miss.

4. Two of the eight violations were suppressed with expiry dates rather than fixed. What makes a violation suppressible, and who decides?

5. The rebuild found a vendor that had corrected history in place. What should Kestrel do about that, given they cannot change the vendor?

6. The non-deterministic dedup had been silently choosing between duplicate records for an unknown period. How would you determine the blast radius after the fact?

7. What is the equivalent of a quarterly rebuild for a system that cannot be rebuilt — a platform whose sources do not retain history? Is the pattern still worth adopting there?