Part II — Storage
Chapters 7–12
Storage decisions are the hardest to reverse.
You can rewrite a transformation in an afternoon. You can swap orchestrators over a sprint, and people do. But the choice of where data physically lives — which system, which format, which partitioning scheme, which layout on disk — propagates into every query written against it, every tool connected to it, and every cost line on the bill. Changing it means a migration, and migrations are measured in quarters.
That asymmetry is why storage comes before ingestion in this book, which is not the order the lifecycle runs in. You should know where data is going before you write the thing that moves it.
Chapter 7 covers relational databases from a data engineer's perspective, which is a different
perspective from an application developer's. You are usually a reader of someone else's
transactional database, and that changes everything: what an index costs you versus costs them, why
SELECT * at 09:00 on Black Friday is a self-inflicted outage, what MVCC means for a long-running
extract, and why money lives in integer cents.
Chapter 8 is the data warehouse: Snowflake, BigQuery, Redshift, and the columnar storage model that makes analytical queries a different problem from transactional ones. The separation of storage from compute is the economic fact that reorganized the entire field, and this chapter is where it gets explained properly.
Chapter 9 is the data lake: object storage, Parquet, and the organizational discipline that separates a lake from a swamp. Object storage is not a filesystem, and most data lake failures trace to treating it like one.
Chapter 10 is the lakehouse — Delta Lake and Apache Iceberg — and the specific problems they solve: atomic commits over object storage, time travel, schema evolution, and concurrent writers. This is the most actively changing area in the field and the chapter is written to survive that.
Chapter 11 is file formats and serialization, and it is more consequential than it sounds. The choice between CSV, JSON, Parquet, Avro, and ORC determines compression, scan cost, schema enforcement, and whether a downstream consumer can read one column without reading forty. This chapter measures rather than asserts.
Chapter 12 covers the stores that are not relational and not analytical: key-value, document, wide-column, time-series, search, and vector. Real platforms have several of these, usually acquired one emergency at a time, and knowing what each is genuinely good at is how you avoid acquiring a seventh unnecessarily.
What you will have built by the end of Part II: Kestrel's source database, seeded and running. An object storage bucket with a deliberate layout. Bronze tables as Delta. And a format decision made from measurement rather than preference.
The single idea to carry out of this part: storage layout is a query optimization made in advance. Partitioning, file size, sort order, and compression are decided when you write and paid for every time anyone reads.