Chapter 30: Data Governance

"We had a catalog with four thousand entries and a search box. When somebody needed to know which table to use, they asked in Slack, because the catalog answered a different question."

Overview

Governance has a worse reputation than any other topic in this book, and the reputation is earned. It is usually experienced as a documentation exercise imposed from outside, producing artifacts nobody reads, enforced by a team nobody asked for.

This chapter takes the opposite starting point. Governance is not a set of artifacts; it is the answers to a small number of questions that people ask constantly and currently answer by interrupting each other:

Which table should I use? · Where did this number come from? · Who owns this? · Who can see it? · Am I allowed to keep it? · Is my definition of "active customer" the same as yours?

Every section here is one of those questions, and the test applied throughout is the one from Chapter 19 §19.9: does the artifact survive, or does it rot? — with the answer depending almost entirely on whether it is a by-product of work someone already does, or a separate task.

Chapter 25 §25.12 measured which tables are actually read; §30.2 is what to do with that. Chapter 31 takes §30.6 and §30.7 into the specifically legal territory.

30.1 Why It Has a Bad Reputation

Three failure patterns, and naming them is most of avoiding them:

Governance as documentation. A quarter is spent filling in descriptions for four thousand tables. Nobody reads them, they are stale within a year, and the exercise is not repeated — which is correctly interpreted as evidence it was not worth doing.

Governance as a gate. A review board between engineers and their work. It reduces the rate of change, which is sometimes the point and is usually not, and it produces the reliable outcome of people routing around it.

Governance as a purchase. A catalog product is bought, populated by a crawler, and contains four thousand entries with technical metadata and no meaning. The search box works; the answers are not there.

📐 Design Decision — the question that separates useful governance from filing

For any governance artifact you are about to create: what question does this answer, who asks it, and how often?

If you cannot name the asker, do not build it. That single test would have prevented every failure pattern above.

The questions that are actually asked, in rough order of frequency at Kestrel, measured by counting a quarter of #data-help messages:

Question	Share	Answered by
"which table should I use for X?"	41%	§30.2 — a catalog with certification
"why is this number different from that one?"	22%	§30.8 — one metric definition
"where did this column come from?"	14%	§30.3 — lineage
"can I get access to X?"	11%	§30.5
"who owns this?"	8%	§30.4
"how long do we keep this?"	4%	§30.7

Forty-one percent is one question, and it is not answered by a description field. It is answered by "this one is certified, that one is deprecated, and here is the difference" — which is a judgment somebody has to make and record, and which no crawler produces.

Measure your own. A quarter of your help channel, categorized, takes an afternoon and it will reorder your priorities.

🎓 Interview Angle — "how would you approach data governance?"

A question that invites a bad answer, because the obvious response is a list of artifacts — catalog, lineage, classification, access reviews — which is exactly the framing that produces the failures above.

A strong answer inverts it:

"I'd start by finding out what questions people currently answer by interrupting each other. At the last place I did this, forty percent of the help channel was one question — which table should I use — and that is not answered by a catalog crawler, because it needs a judgment somebody records."

Then the second half, which is where the experience shows:

"And I'd bias hard toward artifacts that are by-products. Lineage from the code and tests as documentation survive because they cannot drift; a wiki page of table owners rots within a year. The question I'd ask of anything proposed is what keeps this true?"

The likely follow-up is "what about compliance requirements?", and the honest answer earns more credit than a diplomatic one: "some governance work exists to produce an artifact for an auditor and will never be useful internally. I'd do it, keep it cheap, and not describe it as a platform improvement — because calling compliance work engineering is how a team ends up resenting both."

What to avoid saying: that governance is about control, standards, or preventing misuse. All three are sometimes true and all three are how the function gets its reputation, and a candidate who leads with them is describing a gate rather than a service.

30.2 The Catalog

A catalog is a search interface over metadata. Crawlers produce the technical half automatically — tables, columns, types, sizes, row counts — and that half is nearly useless on its own, because it answers "what exists?" and the question is "which should I use?"

Five fields that make an entry usable, and only the first is automatic:

What it is. Auto-collected. Schema, types, size, freshness.

The grain. One row per what? Chapter 6, and it is the single most useful sentence in any catalog entry.

Status. ✅ certified · ⚠️ experimental · 🚫 deprecated, use X instead. This is the 41% answer, and it requires a person.

Owner. §30.4.

Known gotchas. "Excludes cancelled orders." "Wholesale customers are excluded — see fct_order_item_all." "Null before 2025-03." — the things people currently learn by getting a number wrong.

models:
  - name: fct_order_item
    description: >
      One row per order line. Grain: (order_id, line_number).
      CERTIFIED — this is the table to use for revenue by product.
      Excludes cancelled orders. Revenue is NET of returns.
      Wholesale is included; use `is_wholesale` to exclude it.
    meta:
      owner: "#data-eng"
      status: certified
      certified_by: "finance@example.com"
      certified_on: "2026-07-14"

⚠️ Failure Mode — a catalog with no deprecation is a catalog that grows only

Kestrel's catalog had four thousand entries and the team's own estimate was that fewer than three hundred were things anyone should use. The rest were staging models, intermediate models, abandoned experiments, and four generations of the same fact table.

Search returned all of them, ranked by string similarity, which meant the top result for "revenue" was fct_revenue_v2_new_FINAL — deprecated in 2024, still present, still queryable, and still returning numbers.

The fix is not better search. It is that an entry must have a status, and the default status for a model nobody has certified is not "fine."

text certified someone is accountable for this being right ~40 models standard it works; use it; nobody has staked their name ~250 internal an implementation detail; do not query directly ~3,400 deprecated use X instead. Queryable until <date>. ~310

The internal tier is the one that made the catalog usable, and it costs nothing: a staging or intermediate model is marked internal by its directory, automatically, and search excludes it by default. Four thousand results became six hundred.

And deprecated needs a date, or it is a label rather than a plan. Chapter 23 Case Study 2's mute audit, applied to models: a deprecation older than its date is escalated, not renewed.

30.3 Lineage: Three Kinds

"Lineage" names three different things with different costs and different answers, and conflating them is why lineage projects disappoint.

Table-level. This table is built from those tables. Cheap — dbt's manifest has it for free (Chapter 19 §19.9) — and it answers "what breaks if I change this?" and "where could this have come from?"

Column-level. This column is derived from those columns. Considerably more expensive, requiring SQL parsing, and it answers "where does net_revenue_cents actually come from?" — which is the question an auditor asks and a debugging engineer needs.

Run-level. This specific row was produced by that specific run. The most expensive and the most useful during an incident — and it is not a lineage tool at all: it is Chapter 24 Case Study 1's _ingested_at and run_id, and Chapter 27 Case Study 1's _built_by. Three columns.

The ordering is the practical advice. Table-level costs nothing and answers most questions; run-level costs three columns and answers the questions that arise at 05:00; column-level is the expensive one and is usually bought first, because it is the one that demonstrates well.

Buy column-level lineage when you can name the question, and the two that genuinely need it are: "which reports use this PII column?" (Chapter 31) and "an auditor wants the derivation of this figure." Both are real, and neither is "so engineers can understand the models" — for which reading the SQL is faster.

🔎 Read the Plan — what lineage cannot tell you

Lineage is sold as an answer and is a narrowing. Chapter 22 Case Study 2 took three days to diagnose with complete table-level lineage available, because:

Lineage shows where a defect COULD have come from, not where it did. Six upstream sources means six candidates, and the tool cannot rank them.

It stops at your boundary. The supplier's export tool changed its quoting behaviour, which is upstream of every lineage graph anyone has.

It says nothing about why two things are connected. An edge means "reads from." It does not mean the reading is important, correct, or intended — Chapter 19 Case Study 1's hardcoded reference was an edge that should have existed and did not, and lineage cannot show a missing edge.

What makes lineage actually useful is combining it with something else:

• Lineage + freshness → the alert names its six sources and their staleness. A three-day search becomes a six-item checklist, which is Chapter 25 §25.10.
• Lineage + usage (Chapter 25 §25.12) → "this model has nine descendants and none of them is read."
• Lineage + ownership → "this change affects two exposures owned by finance." Chapter 27's PR comment.

On its own, a lineage diagram is a picture. The value is entirely in what you join it to.

30.4 Ownership

The field that makes every other field work, and the one most often filled in uselessly.

Three properties of an owner that is real:

A team, not a person. People leave and change roles; #data-eng survives a reorganization and alice@ does not.

Someone who can act. Chapter 23 §23.12: a test's owner is whoever can fix what it reports. The same applies to a table — the owner of a source table is the producing team, not the data team that ingests it.

Verified, not asserted. ⚠️ An owner field that nobody checks decays, and the decay is invisible:

-- Owners that no longer resolve to anything.
SELECT model, owner FROM catalog
 WHERE owner NOT IN (SELECT name FROM active_teams);

Kestrel's first run of that query returned 34 of 290 owned models — teams that had been renamed, merged, or dissolved. Every one of those models had an owner field that looked filled in.

And the honest problem with ownership, which no amount of process solves: some things genuinely have no owner, because they were built for a purpose that ended and nobody inherited them.

The useful response is a tier, not a fiction:

owner: "#data-eng"                     someone is accountable
owner: "#data-eng (custodial)"         we operate it; we do not know why
owner: null + deprecation date         nobody owns it; it goes away on <date>

The middle tier is the honest one and most catalogs do not have it. Assigning a real owner to something nobody wants produces a fictional owner, which is worse than an explicit custodial, because a fiction is indistinguishable from the real thing.

30.5 Access Control

The mechanics are the easy part. Role-based access control (roles hold grants, users hold roles) covers almost everything; attribute-based control (policies evaluated per row or column) covers masking and row filtering and is Chapter 31's material.

The hard part is that grants accumulate and nothing removes them.

⚠️ Failure Mode — grant sprawl is the alerting ratchet, in permissions

Chapter 25 Case Study 2's mechanism, exactly: every incident adds a grant, and nothing ever removes one.

text "the analyst can't see the new table" → grant added "the migration script needs write access" → grant added, incident over "the vendor's tool needs read on gold" → grant added "just give the BI service account SELECT ANY TABLE, we'll narrow it later"

Each is a reasonable response under time pressure, and the narrowing never happens because narrowing has a cost (something might break) and no benefit anyone can point to.

Kestrel's audit found:

text roles 47 roles with at least one member 31 roles granted to exactly one user 19 ← not roles identities with SELECT on gold 62 identities that had queried gold in 90 days 23 grants added during an incident and never reviewed 14

Nineteen single-user roles is the diagnostic. A role granted to one person is a permission wearing a role's clothes, and it means the role model has stopped describing job functions and started describing individuals — at which point it provides no abstraction and all of the overhead.

Three practices, in order of value:

• Grant to roles that correspond to job functions, never to individuals, and treat a single-member role as a smell to investigate rather than a state to accept.
• Expire incident grants automatically. A grant made during an incident gets 7 days and a calendar reminder. This one change removed 14 of the 47 roles.
• Review against usage, not against intent. The 62-versus-23 gap is the actionable number, and it comes from the same query as Chapter 25 §25.12's.

And the reason to care is not tidiness. It is that Chapter 31's obligations — who can see this personal data — are unanswerable when the answer is "sixty-two identities, of which nobody can account for thirty-nine."

30.6 Classification

What is sensitive, and who decides.

Three tiers cover most organizations, and more than four is usually a sign that classification has become the goal rather than a means:

Two hard parts, and neither is technical:

Who decides? ⚠️ Not the data team. An engineer classifying a column is guessing at a legal and commercial judgment. The classification comes from legal, privacy, or the business owner; the data team implements it — and a platform where engineers classify is a platform where classification reflects engineering intuition.

Classification must attach to the column and propagate. A email column classified confidential in bronze must still be confidential in a gold model that copies it. Manual re-classification at every layer does not survive, which is §30.11's argument arriving early:

-- Propagate, do not re-declare. dbt meta + a check that gold's
-- classification is at least as strict as its inputs'.

And propagation has a rule that surprises people: a derived column can be more sensitive than its inputs, and never less. A hash of an email is still identifying if the input space is small enough to enumerate — which for email addresses it is. An aggregate over few enough rows re-identifies its members, which is why "we only expose totals" is not by itself a control.

So the check is directional:

gold.classification  >=  max(classification of every input column)

Anything that lowers a classification is a deliberate act — an anonymization, a coarsening, a suppression — and it must be reviewed by whoever set the original classification, not by the engineer who wrote the model. Chapter 31 §31.4 is where the techniques for legitimately lowering one live, and §31.3 is why the enumeration attack above works.

30.7 Retention and Deletion

Two obligations that pull in opposite directions, and the tension is real:

Keep it, because analysis wants history and Chapter 26 §26.5 wants recovery windows. Delete it, because Chapter 31's regulations require it and because unnecessary data is liability.

Three things a retention policy needs, and most have the first only:

A period per classification. Restricted: 90 days. Confidential: 3 years. Internal: indefinite.

A mechanism. ⚠️ A policy with no mechanism is a document, and Chapter 9's lifecycle rules, Chapter 10's VACUUM, and a scheduled deletion job are the mechanisms. Chapter 28 Case Study 2 found a lifecycle rule nobody knew about, which is the same problem from the other end.

An exception path. Legal hold, an open dispute, a regulatory investigation. A retention system with no way to suspend deletion for specific records will eventually delete something it should not, and that failure is unrecoverable.

And the one nobody plans for: deletion must reach every copy. The lake, the warehouse, the backups, the BI tool's extract, the analyst's spreadsheet. Chapter 31 §31.5 is where this becomes a legal problem rather than a hygiene one.

30.8 One Definition of a Metric

The 22% question from §30.1: "why is this number different from that one?"

The answer is almost always that there are two definitions, both defensible, in two places:

-- The analytics team's dashboard
active_customer := ordered in the last 90 days

-- The finance team's report
active_customer := ordered in the last 12 months AND not refunded everything

-- The growth team's model
active_customer := any event in the last 30 days

All three are reasonable. None is wrong. And a meeting where two of them are quoted is a meeting about definitions rather than about the business.

A semantic layer — dbt's MetricFlow, Cube, LookML, or a set of certified models — defines the metric once and computes it everywhere. That is the mechanism.

The hard part is not the mechanism. It is that defining a metric once requires somebody to decide, and the three teams above each have a reason. The data team cannot make that decision and should not try — its job is to make the disagreement visible, host the conversation, and implement whatever is agreed.

🏭 From the Pipeline — the audit that finds a second definition

Two definitions of a metric produce two numbers, and the second definition is usually invisible until it appears in a meeting.

The detection is a grep, and it works better than it should:

```bash

Every place a metric-shaped concept is computed independently.

grep -rn "interval '90 days'|last_order_date >" models/ dashboards/ notebooks/ ```

Kestrel found active_customer computed in eleven places, with four distinct definitions. The four were reconciled into one certified model over three weeks — and the three weeks were entirely conversation, not engineering.

Two things that came out of it beyond the definition:

The finance definition won, because it was the one used in a filed report. That is not a technical argument and it was the correct tie-breaker — "which of these has already been said to someone outside the company?" resolves most metric disputes faster than any discussion of merit.

Three of the eleven places were not wrong to differ. A churn model genuinely needs a different activity window from a revenue report. They were renamed — active_90d, active_12m — rather than reconciled, which is the outcome most metric-governance efforts miss: the goal is not one definition, it is no two things sharing a name.

30.9 How People Actually Find Data

They ask a colleague. Every study of this, and every honest look at a help channel, says the same thing — and most catalog projects are built as though the answer were "they search."

This is not a failure of the catalog's search. Asking a person is genuinely better on three dimensions, and a catalog that does not beat it on those dimensions will not be used:

The only column the catalog wins outright is the last one, and scale is the asker's problem only after it becomes the answerer's problem — which is why catalog projects are usually initiated by the person being interrupted.

So the design goal is not to replace the conversation. It is to make the catalog answer the two questions a colleague answers, which is §30.2's status and gotchas fields, and which is a judgment rather than metadata.

📐 Design Decision — instrument the asking, and let it write the catalog

Kestrel's most effective governance mechanism was not a catalog feature. It was a Slack workflow on #data-help, and it works because it makes the answer a by-product (§30.11):

When a question is answered in the channel, the answerer can react with 📗. That posts the question and answer to a queue, and once a week somebody spends twenty minutes turning the queue into description and meta.gotchas on the relevant models.

Three properties make it work:

• The catalog entry is written from a real question, by the person who answered it, in the words the asker used. It is therefore findable by the next person with the same question, which a description written by the model's author usually is not.
• It costs the answerer one click. They have already done the work of answering.
• It produces a ranked backlog for free. The questions asked most often are the entries most worth writing, and nobody had to guess.

In its first six months it produced 84 gotchas across 61 models, and the team's assessment is that these are the most-read text in the catalog. They are also the only part written by someone other than the model's author, which is not a coincidence: an author cannot write down the thing they find obvious.

The measurable effect: repeat questions in the channel — the same question asked twice in a quarter — fell from 31% to 12%. The channel got quieter, which is the outcome the person being interrupted actually wanted, and no catalog feature achieved it.

30.10 Governing Data You Do Not Control

Every organization has data it cannot change: a vendor's export, a partner's feed, an acquired system nobody maintains, a regulator's file format. Standard governance advice assumes you can fix the source, and here you cannot.

Four things you can still do, in increasing order of effort:

Describe it as it is, not as it should be. The catalog entry says "the supplier's sku column is sometimes numeric and sometimes text; see the loader's schema declaration" (Chapter 22 Case Study 2). Documenting a defect you cannot fix is more useful than documenting the intended behaviour.

Wrap it. A staging model that normalizes the source's quirks, and which is the only thing downstream models reference. The quirks are then in one place with a comment, rather than in nine.

Observe a contract. Chapter 17 §17.8's observed contract: status observed, guarantees measured rather than promised, and an explicit note that no agreement exists.

Ask. ⚠️ This is skipped far more often than it should be. Kestrel's supplier-CSV incident (Chapter 22 Case Study 2) was caused by an unannounced change, and the supplier was entirely willing to announce future changes when asked — nobody had asked, because "the vendor won't change anything" was assumed rather than tested.

⚠️ Failure Mode — governing the copy instead of the source

The specific trap: you cannot govern the source, so you govern your copy of it — and the copy acquires a classification, an owner, and a retention policy that the source does not have.

Which produces three divergences, all of them silent:

• Your retention is 3 years; the vendor's is 90 days. A deletion request (Chapter 31) is satisfied in your system and not at the source, and you have told the customer it is done.
• You classify a column confidential; the vendor emails it in a spreadsheet. Your control is real and the exposure is elsewhere.
• You own the copy; nobody owns the flow. When the format changes, the owner of your staging model is accountable for a decision made by someone in another company.

The response is not to give up on governing the copy — that is still worth doing. It is to record the boundary explicitly, as a field:

yaml sources: - name: supplier_inventory meta: external: true upstream_owner: "supplier-b, ops@supplier-b.example" upstream_controls: "none known; no agreement; no change notification" our_controls: "schema declared at read; format assertion; 90d retention" last_verified: "2026-08-19"

upstream_controls: none known is the field that does the work. It converts an assumption into a written statement, and the next person to ask "is this data governed?" gets an accurate answer instead of an inherited belief.

30.11 Governance as a By-Product

The chapter's central claim, and Chapter 19 §19.9's argument generalized:

A governance artifact survives in proportion to how much of it is a by-product of work someone already does. An artifact maintained for its own sake decays, and no amount of process prevents that.

Which produces a ranking of governance mechanisms by durability:

Two design rules follow:

Put metadata next to the thing it describes, in the same repository, in the same file where possible — so it appears in the diff of every change to that thing.

Prefer metadata that fails. ⚠️ A description can be wrong forever; a test cannot. "Excludes cancelled orders" as prose rots; as an assertion it is checked nightly:

- dbt_utils.expression_is_true:
    expression: "status <> 'cancelled'"

The strongest form of documentation in this book is an assertion, and Chapter 23's register is, read another way, a governance artifact that cannot go stale.

30.12 When Governance Is Imposed

Sometimes it arrives from outside — a compliance requirement, an audit finding, a new CISO — and the useful skill is translating a stated requirement into something that both satisfies it and is worth having.

Three translations that usually work:

"All data must be catalogued." → Catalogue what is queried (Chapter 25 §25.12's usage report), and mark the rest internal. Full coverage of an unread asset is not a control.

"Access must be reviewed quarterly." → Review against usage, not against a list. A quarterly attestation where a manager ticks 47 boxes is theatre; a report saying "these 39 identities have not queried this in 90 days" is a decision.

"We need lineage." → Ask which question. §30.3 — table-level is nearly free and answers most of them.

And the honest note: sometimes the requirement is genuinely about producing an artifact for an auditor, and no translation makes it useful internally. Do it, keep it cheap, and do not pretend it is engineering — mislabelling compliance work as platform improvement is how a team ends up resenting both.

💸 Cost Check — what governance costs, and what its absence costs

Governance is the hardest thing in this book to fund, because its benefit is the absence of incidents and its cost is somebody's Tuesday.

text WHAT IT COSTS, at Kestrel's size the catalog: a Markdown file plus catalog_audit.py initial ~3 days maintenance ~2 h / month the quarterly access review ~4 h / quarter the metric audit ~3 h / quarter contract reviews ~2 h / month ─────────── ~5 engineer-days/year, plus 3 to start ~$3,400 - $5,000 / year

text WHAT ITS ABSENCE COST, from this book's own history four definitions of active_customer, 28 months the board saw one of them; nobody knows which reconciliation and remediation ~60 h a grant nobody revoked found in the first access review; unquantifiable a table nobody could find, rebuilt from scratch because the original could not be located ~20 h 3-year documented retention, no lifecycle rule 4.19 TB/year of clickstream retained past policy -> a privacy exposure AND $2,600/year

Roughly 80 engineer-hours plus an unquantified privacy exposure, against 5 days a year of prevention. That ratio is favourable and it is not the argument that works.

The argument that works is §30.1's test, applied to the absence: what decision could you not make? "We could not tell the board which definition they had been shown" is a sentence that produces a budget; "we lack governance maturity" is not.

And the cheapest possible starting point costs almost nothing: the metric audit (§30.13). One grep across the BI layer for the same measure defined twice, run once, presented as a list. It takes an afternoon, it always finds something, and it is the artifact that funds everything else on this page.

🧪 Try It — find your organisation's second definition

An afternoon, and it works at every organisation that has more than one analyst.

```bash

1. Pick a measure everyone quotes. Revenue, active users, churn.

2. Find every place it is DEFINED, not every place it is used.

grep -rniE "active_customer|active_user" \ models/ dashboards/ reports/ notebooks/ --include=".sql" --include=".yml"

3. And the half people forget: the BI tool.

Export its metric or calculated-field definitions and grep those too.

```

Then put the definitions side by side, literally, in one file:

```text A models/marts/dim_customer.sql last_order_at > current_date - 90

B the BI tool's "Active Customers" calculated field COUNTD(customer_id) WHERE order_count > 0 AND last_order_at > TODAY()-180

C the marketing team's notebook any event in the last 30 days -- INCLUDING clickstream, not orders

D the board deck, cell C14 a hard-coded number somebody typed in Q3 2025 ```

Four things to record for each, and the third is the one that turns a curiosity into a finding:

text the SQL, exactly who uses it what number it produces TODAY

Running all four and putting the numbers in a row is the deliverable. "Depending on which definition you use, we have between 412,000 and 1,104,000 active customers" is a sentence that ends a meeting, and it is not an argument — it is four SELECTs.

Then the harder half, which is §30.1's test: pick one. Not "reconcile them" — pick one, name its owner, and write down which alternative you rejected and why (Exercise 6.21's form). The others become derived definitions with different names, or they are deleted.

Expect the meeting to be uncomfortable and expect at least one person to be attached to a definition. That attachment is information: it usually means their definition answers a question the chosen one does not, and that question needs its own metric rather than the same name.

📏 Scale Note — governance's shape changes at two thresholds and only two

Table count is not the variable. The variable is how many people can change something without you knowing, and it moves in steps rather than smoothly.

text one team two+ teams many teams (Kestrel) writing + external ───────────────────────────────────────────────────────────────────────── catalog a Markdown file a catalog product with lineage ownership implicit; ask EXPLICIT, per dataset, in a file the person the tooling reads access review a quarterly a quarterly review with conversation DEFAULT-DENY on expiry metric definitions one dbt project a semantic layer, or four definitions (§30.13) classification a column tag generated, enforced at the by hand contract (ch 17 🔐) the failure mode nobody wrote it nobody KNOWS, and the artifact down that would say is stale

The first threshold is a second team gaining write access, and it is the one Exercise 5.16 names for the catalog. Everything in the left-hand column depends on being able to ask a person, and that stops working the moment the person is in a different team with different priorities.

The second threshold is external producers — an acquisition, a vendor feed, a partner. At that point ownership can no longer be assigned; it can only be observed (Chapter 17 §17.8), and the governance artifacts have to distinguish agreed from observed or they are fiction.

What does not change at either threshold: §30.1's test. What decision does this artifact change? A large organisation with a catalog nobody consults has the same problem as a small one with no catalog, and it has spent considerably more to get there.

The number worth tracking instead of table count: how long it takes to answer "who owns this and what is its grain." Chapter 2's 🧪 measures it. Under two minutes is healthy at any size; over thirty means the artifacts have decayed regardless of what the audit reports.

🔁 Idempotency Check — a catalog that regenerates is a catalog that stays true

Every governance artifact decays, and the ones that do not are the ones that are derived rather than maintained.

text field hand-maintained derived, and idempotent ───────────────────────────────────────────────────────────────────────── owner a name, typed once from CODEOWNERS, per path grain a sentence from the dbt model's `description`, which the author must write anyway classification a tag applied in a UI from the contract (ch 17 🔐) upstream dependencies a diagram from manifest.json freshness "daily" MEASURED, from run records row count a number, from once measured, or omitted last verified a date somebody typed the date the generator ran

The right-hand column regenerates from scratch on every run and produces the same catalog for the same repository — which is idempotency, applied to documentation.

Three consequences.

A derived field cannot go stale silently. If the model's description is deleted, the catalog entry is empty rather than wrong — and empty is a finding that catalog_audit.py reports.

The generator's inputs become the things people maintain, and they are things people maintain anyway: a CODEOWNERS file, a model description, a contract. §30.14's "governance as a by-product" is exactly this, and it is the only version of a governance programme that survives a busy quarter.

And the last row is the one to insist on. last_verified must be the date the generator ran, not a date a human typed — because a human-typed verification date is the field most likely to be wrong and the field most likely to be believed.

The test: delete the catalog and regenerate it. If the result differs from what was there, the difference is either a bug in the generator or something that was being maintained by hand — and both are worth knowing.

🧭 Version Note — the catalog became automatic, and the hard part did not

text era what a catalog was what it could not do ───────────────────────────────────────────────────────────────────────── ~2010 a spreadsheet stay true for a month ~2016 a metadata repository you populated by hand the same, with a UI ~2019 automated crawlers know what anything MEANS ~2022 column-level lineage from query logs same now lineage + usage + an LLM writing descriptions know what anything means, and now it produces PLAUSIBLE descriptions of things it does not understand

The genuinely new capability is column-level lineage derived from query logs, and it is worth paying for. It answers "if I change this column, what breaks" — which is unanswerable by hand and is the highest-value question a catalog can address.

The last row deserves care. A generated description of net_revenue_cents will say "the net revenue in cents", which is true, contentless, and indistinguishable from a description that encodes the four rules of Chapter 38. The field that mattered (§30.2's grain and semantics) now looks populated.

The test from §30.1 applies to generated metadata exactly as it does to hand-written metadata: what decision does it change? A description that restates the column name changes none, and filling the field makes the gap harder to find rather than easier.

The practical rule: let tooling generate the mechanical fields — lineage, usage, freshness, schema — and require a human sentence for grain and semantics. Mark generated fields as generated. A catalog where you cannot tell which fields a person stands behind is a catalog whose most important fields you cannot trust.

🔐 Privacy & Governance — the access review is where governance stops being paperwork

Everything else in this chapter is an artifact. The access review is an action, and it is the one that produces uncomfortable findings on the first run, every time.

text the first review at Kestrel, four engineers, ~60 datasets ───────────────────────────────────────────────────────────────────── grants total 41 grants to a person who had left 3 grants for a migration that finished 5 grants with no identifiable requester 9 service accounts with SELECT on everything 4 grants that had never been exercised (Exercise 25.18) 17 ───────────────────────────────────────────────────────────────────── revoked in the first pass 14 restored within a fortnight 1

One restoration out of fourteen is close to the right error rate. Zero would mean the review had only revoked the obvious ones; a handful would mean it was careless. Say that in advance, because the fear of revoking something needed is what stops a first review happening at all.

The mechanism that makes it sustainable is default-deny on expiry (Exercise 30.12). A grant that is not re-approved lapses; a review whose only outcome is "confirm" is not a review and it will be skipped by the third quarter.

And the query that makes it possible is the join between the grant list and the query log (Exercise 30.7). Seventeen never-exercised grants is not an argument you can have without the data, and with it the conversation is thirty seconds per row.

The one to be careful with is the last category. A grant unexercised in 90 days may be an annual process (Exercise 25.18's lesson). Deprecate rather than revoke: restrict, wait a cycle, then remove — and note that the person who complains is the evidence you needed.

30.13 The Kestrel Platform

🧱 Kestrel Platform — Increment 30: governance from what exists

Nothing in this increment is a new system. Every artifact is derived from the dbt project, the query history, or the Terraform configuration.

text platform/governance/ catalog_audit.py ← unowned, undescribed, ungrained, unread, stale access_review.py ← grants vs usage; single-member roles; incident grants metric_audit.py ← §30.8's grep, generalized models/**/_*.yml ← owner, status, grain, gotchas: next to the model docs/ classification.md ← three tiers, decided by legal, implemented by us retention.md ← period, MECHANISM, and the exception path

Seven things this increment must get right:

1. Measure the questions first. A quarter of #data-help, categorized. §30.1 — it will reorder this list.
2. Four status tiers, and internal assigned automatically by directory. Four thousand search results become six hundred.
3. Every certified model states its grain in one sentence and its known gotchas. §30.2.
4. Ownership is a team, is verified against active teams, and custodial is an allowed value.
5. Access reviewed against usage, single-member roles investigated, and incident grants expire in 7 days automatically.
6. Classification comes from legal and propagates; a gold model is at least as strict as its inputs, checked.
7. Every "gotcha" that can be an assertion is one. §30.11 — a description rots and a test cannot.

The exercise that matters is 30.23(a): categorize a quarter of your help channel before building anything. Kestrel's 41% was one question that no crawler answers, and the increment would have been a different and worse project without that number.

30.14 Summary

Governance has a bad reputation because it is usually documentation with no reader, a gate with no purpose, or a purchase with no meaning.

📐 The test for any governance artifact: what question does it answer, who asks it, and how often? If you cannot name the asker, do not build it. Kestrel measured: 41% of help requests are "which table should I use?" — which no crawler answers, because it needs a judgment somebody records.

A catalog entry needs five fields and only the first is automatic: what it is · the grain · status · owner · known gotchas — the things people currently learn by getting a number wrong.

⚠️ A catalog with no deprecation grows only. Four status tiers, with internal assigned automatically by directory, took Kestrel's search from four thousand results to six hundred. And a deprecation needs a date, or it is a label rather than a plan.

Three kinds of lineage: table-level (nearly free, from the manifest) · column-level (expensive, and what an auditor asks for) · run-level (three columns, and the most useful during an incident).

🔎 Lineage is a narrowing, not an answer. It shows where a defect could have come from, stops at your boundary, and cannot show a missing edge. Its value is in what you join it to: + freshness turns a three-day search into a six-item checklist; + usage finds models with no readers; + ownership tells you who to warn.

An owner must be a team, must be able to act, and must be verified — Kestrel's first check found 34 of 290 owners no longer resolved to anything. And custodial is an honest tier: assigning a real owner to something nobody wants produces a fiction, which is worse.

⚠️ Grant sprawl is Chapter 25's alerting ratchet, in permissions. 62 identities with access, 23 who used it, 14 grants from incidents never reviewed, and 19 single-member roles — a permission wearing a role's clothes. Expire incident grants automatically (this removed 14 of 47 roles) and review against usage, not intent.

Classification: three tiers, decided by legal and implemented by the data team — an engineer classifying a column is guessing at a legal judgment — and it must propagate rather than be re-declared per layer.

A retention policy needs a period, a MECHANISM, and an exception path. A policy with no mechanism is a document; one with no legal-hold path will eventually delete something unrecoverable.

🏭 Two definitions of a metric are invisible until they meet in a meeting. Kestrel found active_customer in eleven places with four definitions; reconciling took three weeks of pure conversation. The tie-breaker that works: which of these has already been said to someone outside the company? And three of the eleven were right to differ — they were renamed, because the goal is not one definition, it is no two things sharing a name.

📐 People find data by asking a colleague, and a colleague wins on every dimension except scale. So instrument the asking: one emoji reaction on an answered question puts it in a weekly queue that becomes description and gotchas. Kestrel got 84 gotchas across 61 models in six months — the most-read text in the catalog, and the only part not written by the model's author, because an author cannot write down what they find obvious. Repeat questions fell 31% → 12%.

⚠️ For data you do not control, do not govern the copy and call it governed. Your retention, your classification, and your owner diverge silently from the source's. Record the boundary as a field — upstream_controls: none known converts an inherited assumption into a written statement. And ask the vendor: Kestrel's supplier was willing to announce changes, and nobody had asked.

A governance artifact survives in proportion to how much of it is a by-product of work someone already does. Put metadata in the same file as the thing it describes, and prefer metadata that fails: a description can be wrong forever, an assertion cannot. Chapter 23's register is a governance artifact that cannot go stale.

When governance is imposed, translate it — catalogue what is queried, review access against usage, ask which lineage question. And when no translation makes it useful, do it cheaply and do not call it platform work.

Chapter 31 takes classification and retention into the territory where getting them wrong is a legal matter rather than an untidy one.

Key terms: technical metadata · business metadata · discoverability · external source boundary · grain · certification · deprecation · table-level / column-level / run-level lineage · steward · custodial ownership · RBAC · ABAC · grant sprawl · single-member role · classification tier · retention · legal hold · semantic layer · metric definition