How to Use This Book

The structure of a chapter

Every chapter is seven files. They are meant to be used differently, and the most common way to get less out of this book than it contains is to read only the first one.

File What it is How to use it
index.md The chapter. 8,000–12,000 words, six to nine numbered sections. Read once, straight through, without stopping to run things. Then go back and run things.
exercises.md 14–20 problems in four graded parts. Part D is always the Kestrel Platform increment. Do Part D even if you skip everything else.
quiz.md Twenty multiple-choice questions with an answer key. Take it before rereading. A wrong answer tells you which section to reread; a reread with no diagnosis is a waste of an evening.
case-study-01.md An engineering situation with real numbers. Read the Background and The Problem, close the file, decide what you would do, then read on.
case-study-02.md A second one, usually a failure. Same. At least one case study per chapter is a thing that went wrong.
key-takeaways.md A one-page operational reference. Print it. This is the page you want open at work, not the chapter.
further-reading.md Annotated sources, tiered. Pick one or two. Reading all of them is not the goal and never was.
code/ Runnable code for the chapter. Run it. Change it. Break it deliberately — most of this book's lessons are about what happens after something breaks.

The four learning paths

Chapter 1 §1.7 carries the full table with time estimates. The short version:

Quick Start — six weeks. Chapters 1, 2, 3, 6, 7, 9, 11, 13, 18, 19, 23, 24, 34, 38. This is the minimum path to being genuinely useful on a data team: you will be able to model a warehouse, land data in it, transform it with dbt, test it, schedule it, and explain the medallion architecture in an interview. You will skip streaming, distributed systems theory, governance, and cost. Come back for those.

The Full Course — fifteen weeks. All forty chapters, in order. This is the university-semester path and the one the instructor companion is built around.

Streaming & Real-Time. Chapters 1–4, 11, 14, 15, 17, 21, 25, 29, 32, 36, 38. For engineers whose problem is latency rather than volume. Chapter 4 is not optional on this path — every hard streaming bug is a distributed systems bug wearing a costume.

Platform & Governance. Chapters 1–5, 9, 10, 12, 24, 26–28, 30, 31, 33, 35, 38. For people building the platform other engineers use, and for people who have to answer to a regulator.

Reading the callouts

Twelve kinds of callout appear throughout. They are not decoration; each is a different kind of claim, and knowing which kind you are reading changes how much you should trust it.

Callout What it means
🏭 From the Pipeline Something that happened in production, with its consequence
⚠️ Failure Mode A specific way this breaks and the symptom you will see first
💸 Cost Check Money arithmetic against the book's frozen price basis
🔁 Idempotency Check The "what if this runs twice" question, asked deliberately
🧪 Try It A hands-on task under ten minutes with a checkable result
📐 Design Decision A real fork in the road, with the losing option named
🔎 Read the Plan An EXPLAIN, DAG, or execution plan you should learn to read
🧱 Kestrel Platform This chapter's increment to the running project
📏 Scale Note What changes at a hundred times the volume
🔐 Privacy & Governance PII, retention, access, lineage
🧭 Version Note An API that has moved, and how to check yours
🎓 Interview Angle A question you will actually be asked

The two to read most carefully are ⚠️ Failure Mode and 📐 Design Decision. Failure modes are the compressed experience of things going wrong; design decisions are where the book commits to an opinion and tells you what it gave up to hold it.

Cross-references

Chapter 14 means the whole chapter. Chapter 14 §14.3 means a specific section. §14.3 inside Chapter 14 means the same section, locally. Appendix E means one of the ten lettered appendices.

Every one of these is checked mechanically by scripts/xref_audit.py, which verifies both that the target exists and — for a curated list of topics — that the cited section actually discusses the thing it is cited for. A reference that resolves to a real section that happens to be about something else is the kind of error that survives every ordinary review, so it gets its own tool.

Running the code

Appendix A is the full setup. The absolute minimum:

python -m venv .venv && source .venv/bin/activate   # Windows: .venv/Scripts/activate
pip install -r requirements.txt
docker compose up -d
python platform/seed/seed_kestrel.py --scale small

--scale small gives you roughly a hundred thousand orders — enough for every exercise, small enough to rebuild in under a minute. --scale full generates Kestrel's actual annual volume and takes a while; you want it exactly twice, in Chapter 21 and Chapter 33, and the book says so when you get there.

Everything in code/ is written to be run directly:

python part-03-ingestion/chapter-13-batch-ingestion/code/extract_postgres.py --help

On the numbers in this book

Every figure is one of four things: a frozen anchor value about Kestrel, arithmetic done on the page from those values, output from code in this repository, or a citation. There is no fifth category. Where a number could not be established one of those four ways, the sentence is qualitative instead.

This matters most for prices. Cloud pricing in this book is US list price, us-east-1, on-demand, at the time of writing, and it is used to teach arithmetic. It will be wrong by the time you read it. That is fine and expected — the skill is the model, not the total. Every cost example shows its working so you can substitute today's prices and get today's answer.

If you find a number in this book that contradicts another number in this book, that is a bug. Please report it. It has happened before in this series and it will happen again.

For instructors

The instructor-guide/ directory has per-chapter teaching notes, five syllabi (one-semester, two-semester, bootcamp, lab course, self-paced), exams with worked solutions, lab handouts, and grading rubrics. Chapter 38's capstone is designed as a final project with a reconciliation result that students compute rather than read — those figures appear nowhere in Chapters 1–37, and the build validator enforces that.

If you get stuck

The three most common places to get stuck, in order:

  1. Docker. Almost always a port conflict or memory limit. Appendix A §A.6 has the specific symptoms and fixes.
  2. Airflow. Install it in its own virtual environment, with the official constraints file. Appendix A §A.4. Do not skip this; sharing an environment with Airflow is a well-known way to lose an evening.
  3. Spark on Windows. It works, but it needs winutils.exe and HADOOP_HOME set. Appendix A §A.5. Or use the Docker image and skip the problem entirely, which is what most people should do.

None of the three is a data engineering problem, which is itself a lesson about the job.