Chapter 17 — Key Takeaways (Schema Evolution and Data Contracts)

The page for writing a contract, and for the conversation with a producer.

The argument

Changes will happen. The question is where the failure lands: at the producer, at write time, as a failed deploy owned by the person making the change — or at the consumer, weeks later, as a wrong number owned by someone who did nothing.

Three parties, one missing artifact: the producer cannot know who reads them · the consumer cannot know when the shape changes · the data carries no statement of intent.

A contract is NOT: a schema · a restriction forbidding change · primarily a technology.

Compatibility

Mode A ___ reader can read ___ data Upgrade first
Backward new reader, old data consumers ← default; what replay requires
Forward old reader, new data producers
Full both either
Change Backward Forward Full
Add optional field ✅ ✅ ✅
Remove optional field ❌ ✅ ❌
Widen type (int→long) ✅ ❌ ❌
Rename ❌ ❌ ❌
Reorder ✅ ✅ ✅

⚠️ Adding an enum value is compatible on paper and frequently breaking in practice — the most common "it should have worked" registry incident. Use the TRANSITIVE variant if consumers may read arbitrarily old data. For a replayable log, they can.

The registry

producer ──register──▶ registry ──┬─ compatible → id 4182
                                  └─ NOT → 409 CONFLICT   ← THE ENTIRE VALUE

The bad data never exists. Compare: without one, the change succeeds and surfaces in a consumer weeks later.

  • Messages carry an id, not a schema — 5 bytes vs. a kilobyte, at 14M events/day
  • ⚠️ It is a runtime dependency of every consumer. A registry outage during a cold-cache restart is a real failure — run it with database-level availability
  • It enforces one field of the contract

The seven things beyond the schema

  1. Ownership — a team and a monitored channel
  2. Consumers ← solves the producer's central problem; being listed is what entitles you to notice
  3. Schema (format, subject, compatibility mode)
  4. Semantics ← what the fields MEAN. Catches sign inversions and redefined timestamps — changes with NO schema change
  5. Guarantees — freshness, completeness, ordering, volume, availability
  6. Change policy — how change happens, not whether
  7. Validation — what is checked, where

Four enforcement points

Point Catches Notes
Producer CI incompatible schema before merge cheapest and most valuable
Producer runtime code paths CI missed
Consumer runtime a producer who bypassed it; sources you do not own build this first
Scheduled, vs. guarantees a compliant producer that stopped or halved volume the only point that catches this

Build consumer validation first — you can do it unilaterally, this afternoon, without anyone's agreement, and it generates the evidence that makes the producer conversation possible.

Versioning

Change Bump
Add optional field PATCH
Add field to adopt; clarify semantics MINOR
Remove, rename, retype, change meaning MAJOR = a new topic, not a modification

Expand-contract: ① add alongside, both populated ② consumers migrate at their own pace ③ remove — and the consumer list is what makes step 3 a fact rather than a guess.

The social half — where contracts actually fail

Failure Fix
Written once, never updated lives in the producer's repo; CI fails without it
Nobody knows it exists discoverable from the data — catalog, registry, dbt docs
The producer sees it as an imposition frame it as what they already provide — the consumer list is THEIR benefit
Violations have no consequence a violation is an incident with a retrospective

Sources you do not control

Write it anyway, status: observed. It records what you measured, defines "broken" so it is a check rather than a feeling, and generates evidence.

Two fields make a violation log persuasive rather than accusatory:

  • their_cost_hours — the producer had never connected your questions to their own changes. This is the number that moves the conversation.
  • their_change_was_reasonable — 17 of 19. Keeps the log honest and removes the defensive shape before it starts.

⚠️ An observed contract is a hypothesis. Kestrel's was materially wrong for eight months. Review it quarterly whether or not the producer participates.

Two rules worth carrying

ELSE in a CASE over a categorical column is a decision to be silently wrong about values that do not exist yet. Enumerate every value, omit the ELSE, and let the null fail a not_null test. Same question for every COALESCE default and every except: pass: is this producing an answer that looks right?

A notice is not a control. A process whose only enforcement is someone reading a message will fail, and the failure looks like: everyone compliant, nobody at fault, the outcome wrong. Ask of any agreed process: what fails if nobody reads it?

The nine lines that close the enum gap

SELECT status, COUNT(*) FROM bronze.orders
 WHERE status NOT IN ('pending','paid','picked','shipped',
                      'delivered','cancelled','refunded')
 GROUP BY 1;      -- expect zero rows

Generate it from the contract, so the enumeration lives in one place and a contract change appears as a reviewable PR diff — which turns the notice into a code review.

When a contract is NOT worth it

One team, both sides · genuinely exploratory data · no enforcement point available anywhere · a tolerant consumer and a daily-changing producer.

Not on the list: a different team · an external producer · a producer who will not agree. All three call for an observed contract.