Key Takeaways: Event-Driven Architecture

The one thing

Chapters 14, 15, and 29 treated the log as transport for data whose real home was a table. This chapter asks what happens when the log is the home and every table is a view of it — and answers with four measurements rather than an argument.

Three things called "event-driven"

⚠️ Adopting sourcing's vocabulary with notification's design creates expectations the architecture cannot meet — "we can replay," "we have full history," "state is a fold." None is true, and the failure surfaces eighteen months later when somebody asks for a retroactive projection.

Vocabulary discipline is free. Call it what it is.

What an event is

Past tense — OrderShipped, not ShipOrder. A command can be rejected; an event has happened.

Immutable — corrected by appending a compensating event, never by editing.

Self-contained enough — §36.8's hardest question.

Versioned within its aggregate — which is what makes ordering checkable rather than assumed.

Never a generic OrderUpdated carrying a diff. That is a row update wearing an event's clothes, and you cannot tell from it what happened or why.

The dual write

⚠️ A commit and a publish are two writes with no transaction between them. The lab: 24 lost per 10,000 at a 0.2% crash rate — deploys, OOM kills, restarts. Kestrel's real incident: 1,420 orders in 90 days, understating revenue by $107,678.60.

"Silent" means precisely that no component has both halves of the picture. The service knows it committed and is dead; Kafka cannot know what it should have received.

Publishing first is worse — a phantom event causes work where a missing one merely under-reports. And you cannot retry your way out of a crash between two non-atomic writes.

📐 The transactional outbox trades loss for duplication, which is the right trade, because an idempotent consumer keyed on (aggregate_id, version) makes a duplicate a no-op and no consumer design makes a missing event a no-op. Enforce it with a shared base class — the only version of "every consumer must" that survives a growing team.

🔎 And reconcile against something independent. Kestrel compared gold to bronze, both built from the event stream, so an order lost before Kafka was absent from both sides and the check passed for two years. A reconciliation is only as good as the independence of its two sides, and convenience selects for dependence — the easiest thing to compare against is already in your warehouse, sharing your warehouse's ancestors.

State is a fold

state = fold(reducer, events). The table is not maintained; it is computed.

A projection is a pure function of the events — which is what makes a rebuild trustworthy, and is the property replay silently depends on.

🏭 The payoff is retroactive projections. Two of the lab's four were computed from history that already existed. Against a state table they are unanswerable, not slow: updated_at was overwritten, and a removed line item left no trace at all.

Kestrel's p90 time to ship took twenty minutes. The alternative was adding a column and waiting a quarter. An event log is an option on questions you have not thought of yet.

🏭 But the benefit that survives a change of team is debugging. Two retroactive projections in two years; the "why is this row in this state" query runs several times a week, by support engineers who do not know what event sourcing is. If the log is not queryable by ordinary people, you have the architecture and not the benefit — project every event into a plain table people can SELECT from.

Designing the event

Carry what the event is about, not the entity's whole state.

Never carry a computed value a consumer could compute differently — cents, not order_total.

Additive changes only, and upcast on read in one place.

📐 The event schema is not the database schema. CDC-as-events makes a column rename a cross-team migration, forces consumers to reconstruct an aggregate you threw away, and — the invisible one — has no place for intent. A row going to cancelled does not say who cancelled it, and you discover that when somebody asks a question that needs it, years of history later.

CDC is an excellent way to replicate a database and a poor way to publish a domain event. Kestrel uses CDC for bronze and the outbox for domain events, deliberately as two streams.

Ordering

🔎 Check the algebra before buying the infrastructure.

sum, count, min, max, set union      commutative     no ordering needed
last-write-wins, first-write-wins    NOT             ordering, or a version
a state machine, a running total     NOT             ordering

Shuffle the stream: every sum is exactly correct and 49.0% of statuses are wrong.

You need per-aggregate ordering, not global — partition by order_id. And for non-commutative projections, a stored last_version_seen removes the need for ordering entirely, in six lines.

📐 Partitioning and version checking are complementary, not alternatives. Partitioning is a performance measure; version checking is the correctness property, surviving rebalances, replays, and producer retries. A team that adopts only partitioning is correct until the first rebalance.

Kestrel withdrew a two-quarter global-ordering proposal that none of its four projections needed — and kept one component from it: the gap detector, with new semantics ("an event is missing" rather than "buffer"), which caught a genuine loss in eleven minutes.

Redo the analysis when a projection is added. One line in a PR template.

Replay

🧭 A capability you never exercise is a capability you do not have. Three real failures:

• An old event type nobody handles — silently ignored, correct then, wrong on replay.
• A projection that calls an external service — producing today's enrichment for 2024's events. A projection must be a pure function of the events.
• A schema version nobody remembered.

All three are found only by replaying. Replay quarterly into a scratch schema and diff.

What it costs

💸 The storage objection is not an objection. 10× the state table and $0.27 a month — 30.5 million events, 13 GB, against a storage line that is 2.1% of the bill.

The real costs are engineering time: idempotent consumers forever · the outbox insert the application team must write · projections that are code and can be wrong since written · debugging that becomes a fold. Argue the trade there — and do the storage arithmetic in front of anyone who raises it, because it takes a minute and removes the objection permanently.

Volume decides whether storage is in the conversation. Orders: 13 GB. Clickstream at 14M events a day: 4.19 TB a year.

Retention

🔐 An immutable log and an erasure obligation are in direct conflict.

• Crypto-shred — per-subject keys; delete the key; the log is untouched and unreadable. Best available, substantial commitment.
• Snapshot and truncate — honest, and you lose history before the boundary.
• Rewriting the log — dishonest, breaks immutability and every subsequent version number, and is what teams do under a statutory deadline when they have not planned for it.

The decision that makes either honest option available: keep personal data out of the payload. Carry customer_id, not the address.

And compaction is not retention. A compacted topic keeps the latest event per key, which destroys an event-sourced log, because the earlier events are the data.

When not to

Nobody needs history — the log's option value is zero and its costs are not.

No aggregate boundary — nothing for events to be about, and the version becomes a contention point.

Your team will not maintain projections — four projections is four things that can be wrong.

Ad hoc query is the access pattern — you will build the state table anyway, which is fine and means running both.

The partial adoption is usually right: events for aggregates with a lifecycle, tables for everything else. Kestrel event-sources orders and not the product catalog — and the catalog is the larger table.

The code

code/event_lab.py — the dual write measured against the outbox, four projections including two built retroactively, a shuffle test that separates commutative reducers from order-dependent ones, and a storage comparison. Thirty-nine self-checks, including the cross-check that the state fold and the revenue projection agree. --demo runs all four.