Prerequisites

This book teaches every data engineering concept and tool from scratch. It does not teach programming, SQL, or the command line from scratch. Here is exactly what you need, with a self-check for each, and what to do if a self-check goes badly.

Python — solid, not expert

You should be able to read and write Python without looking up syntax. Specifically: functions with arguments and return values, list and dict comprehensions, classes (defining one, not designing a hierarchy), exceptions and try/except/finally, context managers (with), decorators as users of them, virtual environments, and pip.

Type hints appear on every function signature in this book. If you have not used them, you will pick them up by reading — they change nothing about how the code runs.

Self-check. Read this and say what it prints, and why it is not what a newcomer expects:

from contextlib import contextmanager

@contextmanager
def batch(name: str):
    print(f"begin {name}")
    try:
        yield []
    finally:
        print(f"end {name}")

with batch("orders") as rows:
    rows.append(1)
    raise ValueError("bad row")

If you can explain that end orders prints before the traceback, and why that ordering is the entire reason context managers exist for pipeline code, you are ready.

If not: any solid introductory Python course. The DataField Python for Business for Beginners covers everything above except decorators.

SQL — comfortable with the basics

SELECT, WHERE, JOIN (inner, left, and knowing the difference), GROUP BY with aggregates, HAVING, subqueries, and ORDER BY/LIMIT. You should know what a primary key and a foreign key are and roughly what an index does.

You do not need window functions, CTEs, recursive queries, or pivots. Chapter 18 teaches all of those properly, from the assumption that you have never used them.

Self-check. Given orders(order_id, customer_id, placed_at, status) and order_items(order_item_id, order_id, product_id, quantity, unit_price_cents), write the query for "total revenue per customer in 2025, highest first, customers with at least two orders only."

If you wrote a JOIN, a GROUP BY, a HAVING COUNT(DISTINCT order_id) >= 2, and remembered that SUM(quantity * unit_price_cents) is in cents, you are ready. If you put the COUNT condition in the WHERE clause, review HAVING before Chapter 18.

If not: the DataField Database Fundamentals book, or any introductory SQL course. Appendix B is a reference, not a tutorial — it assumes you already have the basics.

The command line — functional

You should be able to navigate directories, run a program with arguments, redirect output to a file, pipe between commands, set an environment variable, and read an error message without panic. cd, ls, cat, grep, curl, |, >, export.

Bash is a secondary language in this book: orchestration scripts, Docker commands, and CLI tooling. Nothing exotic. Windows users can use Git Bash, WSL, or PowerShell — the book notes where the commands differ, and Appendix A covers Windows specifics in detail.

Self-check. Say what this does without running it:

grep -c "ERROR" logs/*.log | sort -t: -k2 -rn | head -3

Databases — conceptual

You should know that a database has tables, that tables have rows and columns, that queries return result sets, and roughly what a transaction is. You do not need to know how a B-tree works, what MVCC is, or how a query planner chooses a join order. Chapter 7 covers what a data engineer needs of all three.

Cloud — "I have used S3" is enough

Genuinely. If you know that S3 stores objects in buckets and that you access them over HTTP with credentials, you have enough. If you have never touched a cloud platform at all, that is also fine: this entire book runs locally against MinIO, which speaks the S3 API, and the boto3 code is identical.

What you should not do is buy a cloud account to work through this book. You do not need one. The places where the cloud genuinely differs from the local stand-in are called out explicitly.

What you do NOT need

• Prior data engineering experience. None. That is the point.
• Math beyond arithmetic. MATH_INTENSITY for this book is low. You will multiply row counts by byte sizes, divide by throughput, and compute percentages. There is no calculus, no linear algebra, no statistics beyond a percentile. Chapter 4's distributed systems material is conceptual, not formal.
• A big machine. 16 GB of RAM is comfortable; 8 GB works if you run the Docker services in the groups Appendix A describes rather than all at once. Spark is the only genuinely hungry component and Chapter 21 tells you how to size it down.
• A cloud account, a credit card, or a paid tool. Every piece of software in this book is free and open source, or has a functionally complete free local mode.
• Java. Kafka and Spark are written in it. You will never write a line of it. You will, occasionally, read a Java stack trace, and Chapter 21 §21.8 teaches you how to find the one useful line in a hundred.

Software you will install

Appendix A walks through all of it. Summary:

Total disk footprint of the images and generated data at --scale small is around 12 GB.

A note on version drift

This book pins exact versions in requirements.txt and states them in Chapter 1. Data tooling moves fast enough that some of them will be behind by the time you read this. That is expected and mostly harmless — the code is written to survive minor upgrades, and every chapter that shows an API likely to have moved carries a 🧭 Version Note telling you how to check yours.

The one place to be careful is Airflow, which had a major version boundary that changed import paths and some scheduling semantics. Chapter 24 teaches 2.10.5 and has an explicit section on what Airflow 3 changed and how to tell which you are on. Install it in its own virtual environment regardless of version.

If something in the book does not work with your version, the most useful thing you can do is open an issue with the version, the command, and the error. Version-drift corrections are the single most valuable contribution to a book like this one.