Part VI — Advanced Topics
Chapters 29–33
By this point the Kestrel platform works. This part is about the five things that turn a working platform into one a company can grow on, sell against, and survive an audit with.
They have something in common that is easy to miss: each of them is a constraint that arrives from outside engineering. Latency requirements come from the business. Governance comes from compliance, security, and the simple fact that nobody can find anything. Privacy comes from regulation with statutory deadlines. Feature stores come from a machine learning team whose needs are genuinely different from an analyst's. Cost comes from finance, usually abruptly, usually in a meeting.
Engineers who treat these as impositions build platforms that fight them. Engineers who treat them as requirements build platforms that absorb them. The difference is mostly a matter of when you started thinking about it.
Chapter 29 is real-time and streaming architecture: Lambda, Kappa, the spectrum between batch and streaming, event time versus processing time, watermarks, windowing, and the honest accounting of what streaming costs to operate. Most systems described as streaming should be micro-batch, and this chapter says when yours should not be.
Chapter 30 is data governance: catalogs, lineage, access control, ownership, and stewardship. The unifying question is whether anyone can answer "where did this column come from, what does it mean, and who do I ask" without a meeting.
Chapter 31 is privacy engineering: GDPR and CCPA as technical requirements rather than legal abstractions. PII detection, classification, minimization, pseudonymization, anonymization and its limits, retention, and the deletion request that has to reach every copy of a row — including the ones in a lake with no index.
Chapter 32 is ML engineering and feature stores: what a machine learning team actually needs from you, why training/serving skew is a data engineering failure rather than a modeling one, point-in-time correctness, and how a feature store differs from a warehouse table.
Chapter 33 is cloud cost optimization, and it is the chapter this book has been building toward since the $3,840 Spark job in Chapter 1. Where the money goes, how to attribute it, the specific optimizations ranked by payoff, and how to build the habit of computing a cost before shipping a query rather than after receiving a bill.
What you will have built by the end of Part VI: a streaming path alongside the batch one with its correctness properties stated, catalog entries and a lineage graph for the whole platform, PII classification with a working deletion handler, a point-in-time-correct feature view for the data science team, and a complete cost model for the platform with per-component attribution.
The single idea to carry out of this part: none of these five can be added convincingly at the end. Each one is cheap if the platform was built expecting it and expensive if it was not, which is why they appear as a part rather than as an appendix.