Appendix A: Environment Setup

Everything in this book runs on a laptop. No cloud account, no credit card, and no managed service is required to complete any chapter including the capstone.

The sandbox is four containers and about 2 GB of disk. It stands in for the production stack the book teaches, and the substitutions are deliberate: the code you write against them is unmodified when pointed at the real thing.

Plus Python 3.11+ and the pinned packages in §A.4.

A.1 Prerequisites

# verify what you have
docker --version          # 24.0 or newer
docker compose version    # v2; note the space, not docker-compose
python --version          # 3.11 or newer
git --version

Disk: about 6 GB total — 2 GB of images, 1 GB of Python packages, and room for the generated data.

Memory: 8 GB works. 16 GB is comfortable, and Chapter 22's benchmark is more interesting with it because you can see where pandas stops fitting.

Windows: everything here works under WSL2 and under Git Bash. PowerShell users: the paths in this book use forward slashes, which Python accepts on Windows; the shell commands assume a POSIX shell.

A.2 The Compose File

# docker-compose.yml
services:
  postgres:
    image: postgres:16.4
    environment:
      POSTGRES_USER: kestrel
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
      POSTGRES_DB: kestrel
    command:
      - "postgres"
      - "-c"
      - "wal_level=logical"          # required for Chapter 14's CDC
      - "-c"
      - "max_replication_slots=4"
      - "-c"
      - "shared_preload_libraries=pg_stat_statements"
    ports: ["5432:5432"]
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./sandbox/init:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kestrel -d kestrel"]
      interval: 5s
      retries: 10

  minio:
    image: minio/minio:RELEASE.2024-09-13T20-26-02Z
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:?set MINIO_ROOT_USER in .env}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?set MINIO_ROOT_PASSWORD in .env}
    ports: ["9000:9000", "9001:9001"]
    volumes:
      - miniodata:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      retries: 10

  redpanda:
    image: redpandadata/redpanda:v24.2.4
    command:
      - redpanda start
      - --overprovisioned
      - --smp 1
      - --memory 1G
      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    ports: ["9092:9092", "9644:9644"]
    volumes:
      - rpdata:/var/lib/redpanda/data

volumes:
  pgdata:
  miniodata:
  rpdata:

Why ${VAR:?message} rather than a default. Compose fails with your message if the variable is unset, so a missing credential is a startup error rather than a container running with minioadmin. validate.py fails the build on a hardcoded credential anywhere in this book, and this is the pattern it expects.

Create .env beside the compose file, and add it to .gitignore:

POSTGRES_PASSWORD=$(openssl rand -hex 16)
MINIO_ROOT_USER=kestrel
MINIO_ROOT_PASSWORD=$(openssl rand -hex 16)

A.3 Bringing It Up

docker compose up -d
docker compose ps                    # all should be healthy
docker compose logs -f postgres      # ctrl-c to detach

# the four buckets Chapter 5 section 5.9 specifies
docker compose exec minio sh -c '
  mc alias set local http://localhost:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" &&
  mc mb -p local/bronze local/silver local/gold local/scratch'

Verify each one:

psql "postgresql://kestrel@localhost:5432/kestrel" -c "SELECT version();"
curl -s localhost:9000/minio/health/live && echo "  minio ok"
docker compose exec redpanda rpk cluster info
python -c "import duckdb; print(duckdb.sql('SELECT 42 AS answer'))"

Tear down, keeping data: docker compose down Tear down, destroying data: docker compose down -v

A.4 Python Environment

python -m venv .venv
source .venv/bin/activate            # Windows: .venv/Scripts/activate
pip install -r requirements.txt

requirements.txt, pinned — the versions this book was written and tested against:

apache-airflow==2.10.5
dbt-core==1.9.1
dbt-duckdb==1.9.1
pyspark==3.5.3
pandas==2.2.3
polars==1.17.1
duckdb==1.1.3
pyarrow==17.0.0
psycopg[binary]==3.2.3
boto3==1.35.36
confluent-kafka==2.5.3
great-expectations==0.18.22
requests==2.32.3
pytest==8.3.3

🧭 On the pins. Every version above is a version this book's code was run against. Newer will usually work and the failures are specific: pandas 3.x changes several defaults Chapter 22 relies on for its measurements, and Airflow 3.x renames concepts Chapter 24 teaches (see §D.11). If you are learning, pin. If you are building, upgrade deliberately and read the changelog.

Airflow needs its own constraint file, because it pins transitive dependencies aggressively:

AF=2.10.5; PY=$(python -c 'import sys;print(f"{sys.version_info.major}.{sys.version_info.minor}")')
pip install "apache-airflow==${AF}" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AF}/constraints-${PY}.txt"

A.5 Seeding the Kestrel Data

python sandbox/seed.py --orders 2400000 --days 365 --seed 1

Generates the frozen anchors — 2.4M orders, 6,575/day, AOV $75.83, 2.70 lines per order, and a Black Friday at 6.28× the annual average day. Takes about four minutes and produces roughly 900 MB in Postgres.

A smaller run for a laptop under pressure:

python sandbox/seed.py --orders 120000 --days 60 --seed 1     # ~40 MB, 20 seconds

Every ratio is preserved at the smaller size, so the chapters' arithmetic still works; only the absolute figures change. The capstone requires the full set, because its sealed figures are for a specific month.

A.6 What Each Chapter Needs

Spark in local mode needs a JDK: 17 for Spark 3.5.x. java -version should print 17.

A.7 Troubleshooting

Postgres will not start after a config change. The volume holds the old data directory. docker compose down -v and re-seed, or edit postgresql.conf inside the volume.

MinIO buckets disappear. You ran down -v. They are recreated by the mc mb command in §A.3; put it in a script.

Redpanda exits immediately. Almost always memory. Lower --memory to 512M, or raise Docker Desktop's allocation.

Spark: java.lang.UnsupportedClassVersionError. Wrong JDK. Spark 3.5 wants 17.

Spark on Windows: HADOOP_HOME unset. Either run under WSL2 or install winutils.exe for the matching Hadoop version. WSL2 is much less trouble.

dbt cannot find the DuckDB file. profiles.yml paths are relative to where you invoke dbt, not to the project directory. Use an absolute path while learning.

Airflow DAG does not appear. Check dags_folder in airflow.cfg, then airflow dags list-import-errors — which is the command that tells you about the exception the UI is silently swallowing.

docker compose says docker-compose not found. Compose v2 is a Docker subcommand: a space, not a hyphen.

A.8 Running the Book's Code

Every chapter's code/ directory is standalone and dependency-free unless it says otherwise:

cd part-06-advanced-topics/chapter-31-privacy-engineering/code
python pii_scan.py --self-check     # 80 assertions
python pii_scan.py --demo

Run --self-check first, always. It asserts every figure the chapter prints against the fixture, so a passing self-check means the chapter's numbers describe the code in front of you. A failing one on an unmodified file is a bug worth reporting.

Verify the whole book:

python scripts/validate.py          # structure, anchors, sealed figures
python scripts/xref_audit.py        # every cross-reference resolves
for f in part-*/chapter-*/code/*.py; do python "$f" --self-check; done