Chapter 1 — Key Takeaways (What Is Data Engineering?)

The one-page orientation. Come back to this when you need to explain the job to someone — a manager, a recruiter, a data scientist who thinks you build dashboards.

The one-sentence definition

Data engineering is moving data from where it is produced to where it is used, at a scale and reliability that lets someone downstream treat it as true.

The last clause is the engineering problem. Anyone can move data.

The six artifacts you produce

Produce only #1 and #4 and you have built a machine that moves bytes. That is not the job.

The asymmetry that explains everything

software service fails  ──▶  exception, 500, red graph, page      ──▶  found in seconds
data pipeline fails     ──▶  green DAG, populated table, chart    ──▶  found in a month
                             …and the numbers are wrong

"Did it run?" is the wrong question. "Is the output what it should be?" is the right one — and answering it requires deciding, in advance and in writing, what it should be.

The four questions to ask of any pipeline

1. What happens if this runs twice? (Idempotency — Ch. 20 §20.3)
2. What is the declared grain, and what test enforces it? (Ch. 23 §23.4)
3. How would I know if this silently produced wrong data? (Ch. 25)
4. What does this cost, per run and per year? (Ch. 33)

Idempotency, in one comparison

-- NOT idempotent: every run adds rows.  31 runs -> 11.4% revenue inflation.
INSERT INTO fct_order_item SELECT ... WHERE placed_at >= CURRENT_DATE - 3;

-- Idempotent: any number of runs converges to the same state.
BEGIN;
  DELETE FROM fct_order_item WHERE placed_at >= CURRENT_DATE - 3;
  INSERT INTO fct_order_item SELECT ... WHERE placed_at >= CURRENT_DATE - 3;
COMMIT;

Four ways to get it: delete-insert, merge/upsert on a natural key, partition replacement, content-addressed writes. Chapter 20 §20.3.

Who owns a wrong number

A system can be perfectly up and completely wrong. That case is yours.

The eras, and what survived each

Tools change; the lifecycle does not. Generate → ingest → store → transform → serve.

Kestrel Supply Co. — the numbers to remember

The two numbers that drive architecture: the 6.28× peak ratio (elasticity) and the four-person team (can four people run this at 3am?).

The two framing incidents

Duplicate rows. Backfill with no DELETE, scheduled nightly, ran 31 days, inflated revenue 11.4%. Six monitoring checks, all green, all measuring the pipeline. Found by a finance reconciliation, diagnosed in 90 seconds on day 13 by asking whether the primary key was unique. → Test the data. Declare the grain. Ask what happens if it runs twice.

The $3,840 job. WHERE CAST(event_ts AS DATE) = '...' on a table partitioned by event_date. No pruning → 4.2 TB scanned instead of 34 GB.

before:  160 executors × 10.0 h × $2.400 = $3,840.00 / night  →  $1,401,600 / year
after:    24 executors ×  1.3 h × $2.400 =    $74.88 / night  →     $27,331 / year
                                                   51.3× reduction, $1,374,269 saved

In the plan, the tell is one line: PartitionFilters: []. → Cost is an engineering property, decided in code, in seconds, usually without noticing.

Reading a plan — look here first

FileScan parquet ...
  PartitionFilters: [ ... ]   ← empty means nothing was pruned. Check this FIRST.
  PushedFilters:    [ ... ]   ← empty means the reader read everything and filtered after.
  ReadSchema: struct<...>     ← more columns than you need means no projection pushdown.

Cost arithmetic — the frozen basis

$$\text{job cost} = \text{nodes} \times \text{hours} \times \text{rate}$$

One basis, everywhere. Mixing bases produces two contradictory conclusions from one architecture. Full table in Appendix J.

The numbers rule

Every figure is exactly one of:

1. A frozen anchor figure · 2. Arithmetic shown on the page · 3. Output from code you can run ·
2. A citation.

There is no fifth category. Not "a number I remember." If none applies, write it qualitatively.

Three ways this goes wrong, in increasing order of damage: restating a figure slightly differently elsewhere; re-deriving from a printed, rounded value; reusing a number as a different quantity.

Your learning path

Chapter 38 is on every path. Chapter 4 is mandatory on the streaming path.

The acceptance criterion for everything you build

For any calendar month, net revenue from the gold layer equals net revenue from the source database to the cent — and every difference is explained by a documented, tested rule.

Not "the pipeline runs." The number is right, and you can prove it.