Data Engineering

The Infrastructure of Data

Pipelines, Warehouses, Lakehouses, and the Plumbing Nobody Teaches

DataField.Dev

The foundation volume of the DataField data science sequence

Licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). Code samples additionally licensed under MIT.

Forty chapters · fourteen appendices · one data platform you build yourself

Every data science course begins with a clean CSV.

Somebody put that CSV there.

This book is about that somebody.

First edition.

Kestrel Supply Co., the company whose data platform you build across these forty chapters, is fictional. The failure modes are not.

Cloud prices quoted throughout are US list prices for us-east-1, on-demand, at the time of writing. They will be wrong by the time you read this, which is why every cost example shows its arithmetic rather than only its total.

The complete source — text, code, build scripts, and the instructor companion — lives at github.com/datafield-dev/data-engineering. Corrections are welcome and, in a book this dependent on version-specific behavior, expected.