Part IV — Transformation

Chapters 18–23

Transformation is where raw data becomes something a person can trust.

It is also where most of a data engineer's code lives, where most of the compute bill is spent, and where the silent failures happen. A load either lands rows or does not. A transformation can run perfectly and produce a number that is quietly, confidently wrong — a join that fans out, a filter that drops nulls you needed, a definition that drifted from what the business means.

That is why this part ends with a chapter on data quality rather than beginning with one. Testing makes sense once you know what you are testing.

Chapter 18 is advanced SQL for data engineering: window functions, CTEs, recursive queries, pivots, and the set-based thinking that separates a query that runs in 4 seconds from one that runs in 40 minutes. SQL is the lingua franca of this field and the single highest-return skill in the book.

Chapter 19 is dbt, which changed the field less through technology than through sociology: it made transformations into version-controlled, tested, reviewed, documented software. This chapter covers models, materializations, refs, sources, tests, and docs — and is honest about what dbt is bad at.

Chapter 20 is incremental processing and slowly changing dimensions, and it is the most operationally important chapter in this part. Processing only what changed is how a pipeline stays inside its window as data grows; slowly changing dimensions are how history stays queryable. Both are where idempotency stops being a principle and becomes a specific pattern.

Chapter 21 is Apache Spark: the execution model, the DataFrame API, partitions, shuffles, joins, and reading a physical plan. It is also honest about when you should not use it, which is more often than the industry implies.

Chapter 22 covers single-node Python transformation: pandas, Polars, and DuckDB, with measured guidance on which to reach for. A great deal of work that gets sent to a cluster does not need one, and this chapter gives you the thresholds.

Chapter 23 is data quality: testing the data rather than the pipeline. dbt tests, Great Expectations, the taxonomy of checks worth running, where to run them, and how to calibrate an alert that people will still trust in six months.

What you will have built by the end of Part IV: silver models in SQL, a working dbt project with sources, models, and tests, an incremental fact table and a Type 2 dimension, a Spark sessionizer for the clickstream, a measured comparison of Polars and DuckDB against it, and a test suite that would have caught the duplicate-rows incident on day one.

The single idea to carry out of this part: a transformation is a claim about meaning. Write the claim down, then write the test that enforces it. Code that transforms without a stated claim is just a shape change.

Chapters in This Part