Key Takeaways: Data Governance and Cataloging

The one thing

A governance artifact is worth maintaining only if you can name the decision it changes. Everything else in this chapter follows from that test, and most governance programs fail it on most of their fields.

The catalog

Five fields do the work: description, owner, grain, status, gotchas. A catalog with these five, honestly filled, beats one with forty fields mostly empty. Count the empties and leave them empty — a catalog populated in an afternoon with no gaps is a catalog of guesses.

grain is the field that prevents the most errors and is the one most often missing. "One row per order line, per day, after refunds" answers the question that produces wrong joins.

gotchas is the field that gets read. It is also the only field the model's author reliably cannot write, because an author cannot write down the thing they find obvious.

Four status tiers: certified, standard, internal, deprecated. Assigning internal by directory took Kestrel's search from ~4,000 results to ~600 without anyone making 3,400 decisions — a cheap default you can override beats an expensive judgment nobody makes.

Three kinds of lineage, and they answer different questions. Column-level for impact analysis, table-level for orientation, and the one people actually need: "if this breaks, who notices?" — which is a question about consumers, not about tables.

Ownership

An owner must be a team, able to act, and verified recently. Not the author — authorship is a fact about the past and correlates with none of the three.

34 of Kestrel's 290 owner fields no longer resolved to anyone, and nothing had detected it, because an unverified owner field decays silently and reads as fine.

custodial is an honest tier. Some tables genuinely have no owner who can act on them. Recording that is better than assigning a name that will not answer.

Access

Review against usage, not against a list. 62 identities had access to gold; 23 had used it. The gap is the only actionable number in the exercise.

But the gap is a finding, not a decision. Four of Kestrel's 39 idle identities were legitimately idle — a quarterly analyst, an auditor, two seasonal merchandisers — and an automatic rule would have removed all four. The window must be longer than your slowest legitimate consumer.

An attestation is not a review. The test: could the reviewer's answer have been "no"? Kestrel's review completed on time for eight consecutive quarters and removed nothing, and zero removals across two years is itself the finding nobody was looking at.

Invert the default to revoke — but first make restoration cheap and pre-authorized. Otherwise you have moved the pain rather than reduced it, and the objection to the inversion is correct.

"Narrow it later" never happens. Narrowing has a cost and no visible benefit. Expire incident grants automatically; it removed 14 of Kestrel's 47 roles.

Nineteen single-member roles means the role model describes individuals, not job functions — no abstraction and all of the overhead.

Classification and retention

Three tiers, decided by legal, not by engineering. The data team's job is to make the classification operable, not to choose it.

Classification propagates directionally. A model that reads confidential data is at least as sensitive as its source; you cannot declassify by writing a SELECT. Reduction requires a deliberate, documented transformation — and the "safe" aggregate that identifies one person is the standard failure.

A retention policy needs a period, a mechanism, and an exception path. The mechanism is what is missing: Kestrel had 3-year documented retention on clickstream, 4.19 TB/year of it in S3, and no lifecycle rule. A period with no mechanism is a sentence in a document.

Metrics

The goal is not one definition. It is no two things sharing a name.

Four definitions of active_customer coexisted for twenty-eight months across eleven implementations, producing no error and failing no test — until two people using different ones were in the same room.

The discovery mechanism was a meeting, which means it surfaces late, in public, and framed as somebody having made a mistake. A grep finds it in four minutes, and this is one of the few things worth doing prophylactically — quarterly, on your ten most-quoted metric names.

Reconcile the implementations that answer the same question; rename the ones that do not. Three of eleven were renamed, and forcing them to the certified definition would have degraded a production model. The check: what decision is each number used for?

The tie-breaker that ended a three-week dispute in four minutes: which of these has already been said to someone outside the company? Not an argument about merit, objectively checkable, and it identifies the definition with the highest cost of change.

This defect class is invisible to every other control in this book. gross_margin was well-formed, complete, timely, internally consistent — and wrong for seven months, in a pricing decision. That is the real argument for a semantic layer, not consistency-as-tidiness.

How governance actually survives

People find data by asking a colleague, and a colleague wins on every dimension except scale. So instrument the asking. One emoji reaction on an answered question fed a weekly queue that produced 84 gotchas across 61 models in six months, and cut repeat questions from 31% to 12%.

A governance artifact survives in proportion to how much of it is a by-product of work someone already does. Put metadata in the same file as the thing it describes, and prefer metadata that fails: a description can be wrong forever, an assertion cannot. Chapter 23's register is a governance artifact that cannot go stale.

Every "gotcha" that can be an assertion should be one.

Data you do not control

Do not govern the copy and call it governed. Your retention, classification, and owner diverge silently from the source's, and the divergence appears when a deletion request is satisfied in your system and not at the source.

Record the boundary as a field. upstream_controls: none known converts an inherited assumption into a written statement, which is the whole value.

And ask the vendor. Kestrel's supplier was entirely willing to announce schema changes; nobody had asked, because "the vendor won't change anything" was assumed rather than tested.

When governance is imposed

Satisfy the requirement in a way that leaves something useful behind. The cheapest compliant answer and the useful one usually cost about the same, and only one of them is still there next year.

Say what is a guess. A field populated honestly at scale and a field populated to fill the column look identical in the deliverable and differ completely in value.

The recurring finding, one more time

This is the fourth department in which "measure whether a control is operating, not whether it exists" has appeared — after data quality, alerting, and infrastructure. The access review completed reliably for two years and did nothing. A control that runs on schedule and cannot produce its intended outcome is not a control, and nothing about its completion rate reveals that.

The code

code/catalog_audit.py — catalog completeness, stale owners, grant sprawl, classification directionality, retention mechanisms, and the granted-versus-used gap. Twenty-two self-checks. Run --demo to see eleven findings in a deliberately messy fixture, and Exercise 30.11 to add your own.