Index

Page numbers do not exist in a web book, so this index points at chapters. Generated by scripts/assemble.py from declared key_terms and section headings — add a term to a chapter's front matter to get it in here.

0

0. is this the problem? — 26

1

1. atomic commits — 10
1. compaction — 10
1. ingestion — 5
1. offset pagination — 16
1. row_number — the general answer — 18
1. the consumer rebalance — 4
1. there are no directories — 9
1. what is the deadline? — 26

2

2. cursor / keyset pagination — 16
2. distinct on — postgresql and duckdb — 18
2. log cleanup and checkpoints — 10
2. objects are immutable — 9
2. snapshot isolation — 10
2. storage — 5
2. the straggler — 4
2. where did it stop? — 26

3

3. backpressure, or its absence — 4
3. group by with aggregates — 18
3. operations are billed and rate-limited — 9
3. page-token pagination — 16
3. schema enforcement — 10
3. transformation — 5
3. vacuum — 10
3. which of the four is it? — 26

4

4. distinct on the whole row — 18
4. link-header pagination — 16
4. orchestration — 5
4. the decision: stale, or possibly wrong? — 26
4. the thundering herd — 4
4. time travel — 10

5

5. before recovering: check retention — 26
5. cascading failure — 4
5. data quality — 5
5. efficient row-level deletes and updates — 10

6

6. catalog and governance — 5

7

7. bi and analytics — 5
7. recover — 26

8

8. data impact — 26
8. reverse etl — 5

A

a measured comparison — 22
a week, honestly — 1
a worked system design — 39
abac — 30
absences — 25
absorb before you adopt — 12
acceptance criterion — 38
access control — 30
access paths — 7
accumulating snapshot — 6
accuracy — 23
acks — 15
acks, and what each level costs — 15
action item — 26
actionability — 25
adaptive query execution — 21
additive measure — 6
after image — 14
aggregate — 36
alert fatigue — 25, 26
alerts that get acted on — 25
allowed lateness — 29
alternatives considered — 3
analytics engineering — 1, 40
anomaly detection, honestly — 23
anonymization — 31
anonymization, and why it usually isn't — 31
apache arrow — 22
append-only — 34
apply — 28
approximate nearest neighbour — 12
aqe — 21
architecture decision record — 3
arrow: why they interoperate — 22
artifact — 27
as-of join — 32
asset — 24
at-least-once — 4, 29, 36
at-most-once — 4, 29
atomic commits over object storage — 10
atomic rename — 10
attribution, and the share nobody claims — 33
authentication — 16
auto-suspend — 33
autonomy — 35
avro — 11
avro: row-binary — 11

B

b-tree index — 7
b-tree indexes — 7
backfill — 13, 20, 24, 32, 34
backfill window — 26
backfills — 13
backflow — 34
backoff and jitter — 16
backpressure — 4, 15, 29
backward compatibility — 17
batch — 29
batch ingestion — 13
batch processing — 3
batch scoring — 32
batch versus streaming — 3
before image — 14
behavioral interview — 39
big bang migration — 37
bigquery — 8
binlog — 14
bit packing — 8
bitmap heap scan — 7
blameless postmortem — 26
blast radius — 28, 34
bloat — 7
block compression — 11
blue-green — 27
bottleneck — 35
bounded context — 35
breaking change — 17
bridge table — 6
broadcast join — 21
broker — 15
bronze layer — 34
bronze: what the source sent — 34
bucket — 9
buckets — 9
budget — 28
build cache — 27
build, buy, or do without — 3
burn rate — 26
burnout — 40
bus matrix — 6
business metadata — 30
business rule — 23, 38
buy, build, or do without — 5

C

caching, and why it usually does not help — 21
canary — 25, 27
cap theorem — 4
capstone — 38
cardinality — 25
catalog — 10
catalyst — 21
catchup — 24
catchup and backfill — 24
ccpa — 31
certification — 30
change data capture — 14, 36
change detection — 13
change stream — 12
changing the partition count breaks ordering — 15
chargeback — 33
chasm trap — 6
check strategy — 20
checkpoint — 21, 29
choosing — 11, 22
choosing a mode — 17
choosing the scheme — 9
choosing, and not choosing — 12
chunking — 13
circuit breaker — 16, 23
classification — 30
clear and rerun — 24
clickstream — 1
clock skew — 4
clustering and sort order — the main lever — 8
clustering key — 8
coalesce — 21
codecs — 11
column chunk — 9
column mapping — 10
column-level lineage — 30
columnar — 22
columnar format — 11
columnar storage — 8
command — 36
commit — 10
communications lead — 26
compaction — 9, 10, 36
compatibility: backward, forward, full — 17
compensation, briefly — 40
completeness — 23
compression ratio — 11
compression, and why columns compress so well — 8
compute it — 38
concept drift — 32
concurrency — 24
conformed dimension — 6, 34
conformed dimensions and the bus matrix — 6
connector offset — 14
consensus, briefly, and where you meet it — 4
consent — 31
consent and purpose limitation — 31
consequences — 3
consistency — 23
consistent hashing — 4
consumer — 15
consumer group — 15
consumer lag — 15, 29
consumer rebalance — 4
consumer-driven contract — 17
consumers and consumer groups — 15
consuming a change stream — 14
context — 3
continuous deployment — 27
continuous integration — 27
contract test — 16
contracts for sources you do not control — 17
controller — 31
conway's law — 35
conway's law, and which way the arrow points — 35
copy-on-write — 10, 22
correlated subquery — 18
cost — 1
cost as code — 28
cost attribution — 33
cost is decided in code — 33
cost models, and how not to be surprised — 8
cost per order — 33
cost per pipeline — 33
cost per query — 33
coupling — 3, 35
covering index — 7
cpra — 31
cqrs — 36
credit — 8, 33
crypto-shredding — 31
csv — 11
cte — 18
ctes, and the myth — 18
cursor pagination — 16
cutover — 37
cutover readiness — 37

D

dag — 24
data architecture — 2, 3
data as a product — 35
data catalog — 5, 30
data contract — 17, 30, 35
data docs — 23
data downtime — 23
data drift — 32
data engineering — 1
data engineering and its neighbors — 1
data engineering lifecycle — 2
data impact — 26
data interval — 24
data lake — 1, 3
data lakehouse — 3
data leakage — 32
data management — 2
data mesh — 35
data minimization — 31
data pipeline — 1
data product — 2, 35
data product owner — 35
data quality — 1, 23
data quality as a metric — 23
data residency — 31
data subject — 31
data swamp — 9
data that changes retroactively — 16
data transfer — 33
data warehouse — 1, 3, 8
dataframe api — 21
dataops — 2
dataset — 24
dbt package — 19
dbt snapshots — 20
dead letter — 23
dead letter queue — 15
dead letter queues — 15
debezium — 14
debezium and the event envelope — 14
decide whether to, before deciding how — 20
decision — 3
declarative — 28
decommissioning — 37
deduplication — 18
defer — 19, 27
deferrable operator — 24
definition of done — 38
degenerate dimension — 6
degraded mode — 26
delete+insert — 20
deletes and tombstones — 14
deletion vector — 10, 31
delivery semantics — 4
delivery semantics in practice — 15
delta encoding — 8
delta, iceberg, and hudi — 10
denormalization — 12
dense_rank — 18
deploying the orchestrator — 27
deprecation — 17
depreciation — 40
depth and blast radius — 34
designing the event — 36
dictionary encoding — 8
dictionary page — 11
differential privacy — 31
digest pinning — 28
dimension table — 6
dimension tables — 6
dimensional modeling — 6
discoverability — 30
distribution — 25
distribution test — 23
do you actually have domains? — 35
do you need a dedicated vector database? — 12
do you need spark? — 21
docker — 28
docker compose — 5
docker: the image is the environment — 28
document store — 12
document stores — 12
documentation and the dag as an artifact — 19
domain — 35
domain-oriented ownership — 35
downsampling — 12
drift — 28, 32, 37
drift is inevitable — 28
driver — 21
dsar — 31
dual write — 36
duckdb — 5, 22
duckdb: sql over files — 22
duplicates — 14
duration — 25
duration, and why the threshold is wrong — 25
dynamic allocation — 21
dynamic data masking — 31
dynamic task mapping — 24
dynamodb is not quite a key-value store — 12

E

eager evaluation — 22
egress — 33
elasticity — 3
elt — 1, 3
embedding — 12, 32
enabling team — 35
enforcement — 10
enforcement: four places a contract can bite — 17
entity — 32
entity key — 32
environment — 27
environments from one module — 28
environments, and why staging is hard — 27
ephemeral model — 19
erasure — 31
error budget — 25, 26
error budgets, for data — 26
escalation — 26
etl — 1, 3
etl versus elt — 3
evaluating a tool — 5
event — 36
event carried state transfer — 36
event envelope — 14
event notification — 36
event sourcing — 36
event store — 36
event time — 4, 29
event time versus processing time — 4
event time, processing time, and watermarks — 29
event-driven architecture — 36
eventual consistency — 4, 9
eventual consistency, stated precisely — 4
evolution — 10
exactly-once — 4, 14, 29, 36
exactly-once, and why to be suspicious — 4
executor — 21, 24, 28
expand-contract — 17
expand-contract migration — 17
expectation suite — 23
exponential backoff — 16
exposure — 19
extract manifest — 13
extracting from a key-value store — 12
extracting from each — 12
extracting from files — 13
extracting from legacy systems — 13
extracting without causing an outage — 7
extraction — 12

F

fact table — 6
fact table columns — 6
fact tables — 6
factless fact table — 6
fan trap — 6
feature — 32
feature age — 32
feature age: the number nobody computes — 32
feature freshness — 32
feature store — 32
feature versioning — 32
federated computational governance — 35
file compaction — 33
file sizing for bulk loads — 8
finding 1: gzipped csv beats parquet + snappy — 11
finding 2: sorting made it worse — 11
finding personal data you did not know you had — 31
finops — 33
five ways an api is harder — 16
fixed and variable cost — 33
fixture — 27
fold — 36
footer — 9
format-preserving encryption — 31
forward compatibility — 17
forward fix — 27
four things you can do with a sensitive column — 31
four ways data on-call is different — 26
four ways this is harder — 27
frame clause — 18
freshness — 25
freshness check — 23
from change stream to table — 14
from lake to swamp — 9
from notebook to pipeline — 22
full compatibility — 17
full load — 13
full or incremental — 13
full refresh — 20

G

gaps and islands — 18
gdpr — 31
generate — 2
generate: the stage you do not control — 2
generic test — 19
getting better on purpose — 40
gold layer — 34
gold: business definitions, consumed — 34
governance as a by-product — 30
governing data you do not control — 30
grain — 2, 6
grain test — 23
grant — 30
graph databases — 12
great expectations — 23
gross merchandise value — 1
gross revenue — 38
group coordinator — 15
gui etl — 37
gzip — 11

H

handling deletes — 13
handoff — 26
handover — 38
hard delete — 13
hash diff — 20
heap — 7
heartbeat — 25
high-water mark — 13
hive-style partitioning — 9
hot partition — 4
how a reader skips work — 11
how a reader uses that structure — 9
how a row store reads — 7
how airflow actually breaks — 24
how contract programmes fail — 17
how it grows — 1
how many layers? — 34
how mvcc works — 7
how people actually find data — 30
how this book handles numbers, tools, and cost — 1
how to choose — 3
hugeint — 22

I

idempotency — 1, 4, 13, 20, 24
idempotency key — 16
idempotency, retries, and clearing — 24
idempotency: four strategies, with code — 20
idempotent consumer — 36
idempotent producer — 15
idempotent rebuild — 34
idempotent sink — 29
identifiers — 7
if you do adopt: the order that works — 35
image — 28
immutability — 34
import — 28
in-sync replica — 15
incident commander — 26
incident one: the duplicate rows — 1
incident two: the four-thousand-dollar job — 1
incremental extraction without a change feed — 16
incremental load — 13
incremental migration — 37
incremental model — 20
incremental snapshot — 14
incremental strategies across engines — 20
index scan — 7
indexes from the reader's side — 7
individual contributor — 40
inference — 32
infrastructure as code — 28
ingest — 2
ingest: getting data in — 2
ingestion tool — 5
insert_overwrite — 20
inside a parquet file — 9, 11
instrumenting your own pipelines — 25
integer cents — 7, 22
integration test — 27
interoperability standard — 35
inverse conway maneuver — 35
inverted index — 12
is your central team actually the bottleneck? — 35
is_current — 20
isolation levels, briefly — 7

J

jinja: useful, and then suddenly not — 19
jitter — 16
job — 21
joins in a stream — 29
json and json lines — 11
json lines — 11
jsonb — 7
junk dimension — 6

K

k-anonymity — 31
kafka — 15
kafka connect — 14
kappa architecture — 29
kestrel supply co — 1
kestrel through the lifecycle — 2
kestrel's 6am runbook — 26
kestrel's architecture decision record — 3
kestrel's bronze as delta — 10
kestrel's lake — 9
kestrel's order events — 36
kestrel's readiness, and the honest answer — 35
kestrel's three generators — 2
key — 9
key-value store — 12
key-value stores — 12
know your meter — 33
kubernetes — 28
kubernetes, honestly — 28

L

l-diversity — 31
label — 32
labeling delay — 32
labels are data too — 32
lag — 18
lakehouse — 3, 10
lambda and kappa, with hindsight — 29
lambda architecture — 29
landing raw — 9
late arrivals and the lookback window — 20
late-arriving data — 13
late-arriving dimension — 20
late-arriving dimensions — 20
latency — 3, 29
lateral join — 18
lawful basis — 31
layer — 28
layer boundary — 34
layer drift — 34
layer skipping — 34
laying out a lake — 9
lazy evaluation — 21, 22
lead — 18
leader-follower — 4
leaderless and quorums — 4
least privilege — 30
leaving a tool — 5
legacy system — 37
levelling — 39, 40
levelling, scope, and the offer — 39
lie 3: its granularity is too coarse — 13
lie 4: it moves backwards — 13
lifecycle policy — 9, 33
lineage — 25, 30
lineage as an instrument — 25
lineage depth — 34
lineage: three kinds — 30
link header — 16
lint — 27
liquid clustering — 10
live coding — 39
load-bearing spreadsheet — 37
loading data — 8
log — 25
log compaction — 14, 15
log sequence number — 13, 14
logical date — 24
logical decoding — 14
logical replication — 7
long-running transaction — 7
lookback window — 20
lsn — 14
lz4 — 11

M

making it a habit — 33
managed service — 5
manifest — 9, 10, 19, 27
margin — 25
mart — 19
masking and row-level security — 31
materiality — 38
materialization — 8, 18, 19, 32
materialized view — 8, 29, 36
measuring the rotation itself — 26
measuring: kestrel's clickstream — 11
medallion architecture — 34
memory and cost — 25
memory is the number that decides — 22
memory multiplier — 22
memory, and the four ways a job runs out of it — 21
mentorship — 40
merge — 20
merge-on-read — 10
metadata — 30
metric — 25
metric definition — 30
metrics, logs, and traces — 25
micro-batch — 3, 29
micro-partition — 8
microbatch — 20
migration — 27
minio — 5
ml infrastructure — 40
model registry — 32
modeling kestrel — 6
modern data stack — 5
module — 28
money is integer cents — 7
monitoring — 25
monitoring a cdc pipeline — 14
monitoring a model is monitoring data — 32
monitoring the consumers — 25
multi-stage build — 28
multipart upload — 9
mvcc — 7
mvcc and the long-running transaction — 7

N

narrow and wide — 21
narrow transformation — 21
natural key — 6
negotiating with the source team — 13
nested data, and when to flatten — 11
nested partitioning — 9
net revenue — 38
nine rules a build can enforce — 34
node selection — 19
non-additive measure — 6
non-determinism — 37
normalization — 6
normalization, and why analytics undoes it — 6
nullable dtype — 22
numbers — 1
numeric type — 7

O

oauth — 16
object dtype — 22
object storage — 9
object storage is not a filesystem — 9
observability — 25
observability tool — 5
offer negotiation — 39
offline store — 32
offset — 15
offset pagination — 16
oidc — 27, 28
oltp — 7
on-demand pricing — 8, 33
one definition of a metric — 30
online and offline stores — 32
online inference — 32
online store — 32
onsite loop — 39
opentelemetry — 25
operating a replication slot — 14
operating a streaming pipeline — 29
operating cdc alongside the batch extract — 14
operating kafka — 15
operator — 24, 28
optimistic concurrency — 10
orc — 11
orchestration — 1, 2
orchestrator — 5
ordering — 4, 14
ordering guarantee — 36
ordering: which projections actually care? — 36
out-of-core — 22
output port — 35
ownership — 17, 23, 30

P

pacelc — 4
pacelc, which is the useful version — 4
packages, and the four worth having — 19
page — 7, 9
page token — 16
pages and the heap — 7
pagination — 16
paging — 26
pandas — 22
pandas' four structural problems — 22
parallel run — 37
parallelism and throughput — 13
parquet — 9, 11
parquet and orc: columnar — 11
part vi ends here — 33
part vii ends here — 37
partial failure — 4
partial failure, and why it changes everything — 4
partition — 15, 21
partition alignment — 20
partition assignment — 15
partition by — 18
partition key — 4, 12, 36
partition pruning — 1, 9, 21, 33
partition size — 4
partitioning — 4, 8
partitioning in object storage — 9
partitions: the number that decides everything — 21
periodic snapshot — 6
personal data — 31
personal data in non-production environments — 31
physical replication — 7
pii — 31
pivot — 18
pivot, portably — 18
pivoting and unpivoting — 18
plan — 28
platform engineering — 40
platform team — 35
point-in-time correctness — 32
polars — 22
polars: expressions and laziness — 22
poll loop — 15
polyglot persistence — 12
polyseme — 35
pool — 24
portfolio — 40
portfolio project — 39
postgresql, mysql, and the rest — 7
practical properties — 9
pre-authorization — 26
predicate pushdown — 8, 21, 22, 33
predicate pushdown and row groups — 8
prefix — 9
prefixes — 9
presenting the capstone — 39
privacy as a pipeline stage, not a review — 31
processing time — 4, 29
processor — 31
producer — 15
producer-side validation — 17
producers — 15
profiles.yml — 19
progressive refinement — 34
project layout, and why the boring one wins — 19
projection — 36
projection pushdown — 8, 22
promotion — 27, 40
protocol buffers — 11
provider — 28
pseudonymization — 31
publication — 14
purpose limitation — 31

Q

qualify — 18
quarantine — 23, 34
quarantine: where bad rows live — 34
quasi-identifier — 31
query plan — 7, 22
querying a lake — 9
quorum — 4

R

rank — 18
rank by payoff, not by size — 33
rate limit — 16
rate limits — 16
raw zone — 34
rbac — 30
rdd — 21
re-identification — 31
read replica — 7
read-your-writes — 4
reading a query plan — 7
reading a rejection — 39
reading a transformation's plan — 18
reading files is where it actually breaks — 22
reading the limit — 16
reading the log to debug — 36
reading the table — 10
rebalance — 15
rebalancing — 15
rebuild — 38
reconciliation — 13, 37, 38
reconciling: what "the same" means — 37
recursive cte — 18
recursive ctes — 18
red flag — 39
redshift — 8
ref — 19
referential integrity — 23
refund netting — 38
relevance scoring — 12
repartition — 21
replay — 29, 34
replay window — 16
replication — 4
replication factor — 15
replication lag — 4
replication slot — 7, 14
replication, and which kind cdc needs — 7
reserved instances — 33
residency: where the bytes physically are — 31
respecting it proactively — 16
rest api — 16
restartability — 13
restartable, and why that is not idempotent — 13
result cache — 8
retention — 10, 15, 30, 31
retention and compaction — 15
retention and deletion — 30
retention: how long to keep what — 25
retention: the log that cannot be forever — 36
retroactive change — 16
retryable error — 16
revenue recognition — 38
reverse etl — 2, 5
reverse etl, and the loop that closes — 2
reverse interview — 39
reversible decision — 3
review date — 3
reviewing a data change — 27
right to be forgotten — 31
role explosion — 30
role-playing dimension — 6
rollback — 27, 37
rollback, and what cannot be rolled back — 27
rotation — 26
rotation design — 26
routing — 25
row group — 8, 9
row-binary format — 11
row-level deletes and updates — 10
row-level security — 31
row-oriented storage — 7
row_number — 18
rows versus range — 18
rows versus range — the bug — 18
run-length encoding — 8
runbook — 25, 26, 38
running an incident — 26

S

salting — 21
salting, and the other skew fixes — 21
sampling — 27
savings plan — 33
scan cost — 11
scd type 1 — 20
scd type 2 — 20
scd2 mechanics, and three invariants — 20
scheduler — 24
schema change event — 14
schema changes in a stream — 14
schema drift — 13, 25
schema enforcement — 10
schema enforcement and evolution — 10
schema evolution — 9, 10, 17, 36
schema evolution and replay — 36
schema registry — 11, 15, 17
schema-on-read — 11
schema-on-read versus schema-on-write — 11
schema-on-write — 11
schemas and the registry — 15
scope — 39, 40
scoring a team from outside — 40
search engines — 12
search index — 12
secondary index — 12
secret — 27
secret manager — 28
secrets — 27
security — 2
seed — 19
seeding kestrel — 7
self-serve data platform — 35
semantic layer — 2, 5, 30
semantic versioning — 17
semi-additive measure — 6
semi-structured data — 7
sensitivity tier — 30
sensor — 24
sensors, deferrable operators, and datasets — 24
sequence number — 36
sequential scan — 7
serialization — 11
serve — 2
serve: the only stage the business sees — 2
service level agreement — 17
session window — 29
sessionization — 18
set-based thinking — 18
settingwithcopywarning — 22
seven wastes, sized — 33
severity — 23, 26
shadow and blue-green for models — 27
shadow deploy — 27
shadow running — 37
shadow schema — 17
sharding — 12
showback — 33
shuffle — 21
signal — 39
silver layer — 34
silver: typed, deduplicated, conformed — 34
singular test — 19
sizing partitions — 15
skew — 4, 21
skew is the failure mode — 4
skew: the failure that looks like slowness — 21
sla — 1, 25, 26
sla miss — 24
sla, slo, sli — 26
sli — 26
sliding window — 29
slim ci — 19, 27
slim ci in practice — 27
slim ci, and what it costs not to have it — 19
slo — 25, 26
slot — 8, 33
slowly changing dimension — 20
slowly changing dimensions — 6
slowly changing dimensions, honestly — 20
small file problem — 33
small files problem — 9, 21
snappy — 11
snapshot — 7, 10, 14, 19, 20
snowflake — 8
snowflake schema — 6
snowflake, bigquery, redshift — 8
soft delete — 13
software engineering — 2
sort key — 12
sort order — 11
sort-merge join — 21
source — 19
source fidelity — 34
source freshness — 19
source of truth — 1, 38
source system — 2
specialization — 40
spill — 8, 21
splittable — 11
splitting state, and blast radius — 28
spot instances — 33
sql round — 39
staff engineer — 40
stage — 21
staging — 27, 34
staging model — 19
star — 39
star and snowflake — 6
star schema — 6
starting from nothing: the first ninety days — 17
state comparison — 27
state file — 28
state is a fold — 36
state store — 29
state, and what it costs — 29
statement timeout — 7
statsd — 25
staying current without chasing everything — 40
step 1 — select the business process — 6
step 2 — declare the grain — 6
step 3 — identify the dimensions — 6
step 4 — identify the facts — 6
steward — 30
storage class — 9, 33
storage-compute separation — 8
store — 2
store: the substrate, not a stage — 2
stored procedure — 37
strangler fig — 37
strategy 1 — overwrite the partition — 20
strategy 2 — merge on a unique key — 20
strategy 4 — build aside and swap — 20
stream processing — 3
streaming — 29
streaming execution — 22
structure: recovering from the middle — 24
structured logging — 25
subject access request — 31
surrogate key — 6, 20
surrogate keys — 6
synthetic data — 27
system design interview — 39
system design: what is actually scored — 39

T

table format — 10
tagging — 28, 33
take the inventory before you read anything — 37
take-home — 39
target — 19
target leakage — 32
task — 21, 24
task group — 24
taskflow api — 24
technical currency — 40
technical debt — 37
technical metadata — 30
technical screen — 39
telemetry — 25
telling another team what they cost — 33
terminal error — 16
terraform — 28
terraform: state, plan, apply — 28
testing a dag — 24
testing a python transformation — 22
testing a streaming job — 29
testing against an api you do not control — 16
testing and tuning an incremental model — 20
testing infrastructure — 28
testing the data — 23
testing the pipeline — 23
testing the pipeline versus testing the data — 23
testing without production data — 27
tests belong at different layers — 34
tests: the feature that justifies the tool — 19
text formats — 11
the "do without" column — 5
the acceptance criterion — 1
the access request: export, not delete — 31
the assembly — 38
the behavioral round — 39
the big data era (2005–2015) — 1
the boundary — 28
the boundary people draw in the wrong place — 2
the build that must not be slow — 27
the bus matrix — 6
the bus matrix, condensed — 6
the business, in numbers — 1
the buy/build table, category by category — 5
the calculation — 15
the catalog — 30
the comparison — 10
the comparison that matters — 3, 8
the correctness round — 39
the csv that somebody made — 1
the dba era (1980s–1990s) — 1
the decision — 12
the decision, honestly framed — 5
the definitions that had to be resolved — 6
the differences, and who is wrong — 37
the dimensions — 6
the dual write — 36
the eight categories — 5
the eight common mistakes — 6
the eight fallacies, compressed — 4
the envelope — 14
the etl era (1990s–2000s) — 1
the fact tables — 6
the failure — 7
the failure modes you will actually meet — 4
the first ninety days in a data role — 40
the five change-detection strategies — 13
the five stages, in one sentence each — 2
the four controls to set on day one — 8
the four controls, and all four are required — 14
the four kinds, and what each costs — 37
the four pagination shapes — 16
the four principles, stated fairly — 35
the four rules — 38
the four ways people leave — 40
the four ways updated_at lies — 13
the four-step design process — 6
the frame clause — 18
the functions you will actually use — 18
the general form — 18
the goal — 1
the initial snapshot — 14
the job nobody owns and nobody can read — 37
the kestrel carrier ingester — 16
the kestrel cdc pipeline — 14
the kestrel clickstream pipeline — 15
the kestrel dbt project — 19
the kestrel extractor — 13
the kestrel local stack — 5
the kestrel orders contract — 17
the kestrel platform — 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 34, 36, 37
the kestrel silver models — 18
the last thing — 40
the layers of transformation — 2
the layout — 8
the levels — 40
the margin — 25
the migration that should not happen — 37
the model is a select statement — 19
the model: dag, task, operator, executor — 24
the modern era (2015–now) — 1
the month — 38
the myth — 18
the new shape — 8
the nine formats, placed — 11
the old shape — 8
the one dashboard — 25
the one honest reason to add a store — 12
the optimizations that actually matter — 33
the order — 37
the other three reconciliations — 38
the out-of-core question — 22
the partition key decides everything — 15
the pattern that gets people into trouble — 10
the picture — 2, 3
the pipeline, ordered by yield — 27
the political problem — 37
the postmortem — 26
the pre-flight estimate — 33
the primitive: put-if-absent — 10
the principles that survive — 3
the privacy case — 10
the problem it solves — 35
the problem, stated precisely — 17
the question — 25
the questions you ask them — 39
the reconciliation — 38
the regulations as technical requirements — 31
the retroactive projection — 36
the rules that generalize — 15
the runbook — 26
the same workload, three meters — 33
the scan-cost argument — 11
the schema registry — 17
the self-serve platform — 35
the separation of storage and compute — 8
the seven patterns — 6
the shape of a batch extract — 13
the shape of the loop — 39
the six artifacts — 1
the six assertions that earn their keep — 23
the six dimensions, and which you can test — 23
the six undercurrents — 2
the small files problem — 9
the social half — 17
the source side — 38
the specializations — 40
the spectrum — 29
the spectrum, honestly — 3
the sql round — 39
the storm — 15
the strangler fig — 37
the systems — 1
the take-home, and whether to do it — 39
the team and the constraint — 1
the technique — 18
the term is a problem — 5
the three approaches — 14
the three conditions that decide it — 13
the three kinds of cdc — 14
the three mistakes — 11
the three properties of a safe watermark — 13
the three questions — 20
the three questions that decide it — 3
the three schemes — 4
the three shapes that give it away — 18
the three things it is genuinely good for — 10
the three types — 6
the transaction log — 10
the transformation that will bite you — 2
the transition that matters — 40
the two heartbeats — 15
the two ideas worth stealing — 35
the two models, and what each rewards — 8
the two questions — 11
the two things it is not — 10
the two ways to be wrong — 3
the word "pipeline" hides the work — 1
there is no global clock — 4
third normal form — 6
threads — 19
three things called "event-driven — 36
threshold — 23
throughput — 3
thundering herd — 16
time travel — 10
time, clocks, and ordering — 4
time-series database — 12
time-series databases — 12
timeline — 26
timeliness — 23
timestamp strategy — 20
timestamp with time zone — 7
toil — 26, 40
token bucket — 16
token refresh — 16
tokenization — 31
tombstone — 10, 13, 14, 15
tools — 1
topic — 15
topics, partitions, and offsets — 15
total cost of ownership — 5
trace — 25
trailing median — 25
training/serving skew — 32
transaction boundary — 14
transaction isolation — 7
transaction log — 10
transactional outbox — 36
transactional sink — 29
transform — 2
transform: making it mean something — 2
transformation framework — 5
transitive compatibility — 17
trigger — 24
ttl — 12
tumbling window — 29
two incidents that frame this book — 1
types and nulls: where the three disagree — 22
types that matter — 7

U

udfs: the cost that is not in the plan — 21
undercurrent — 2
undocumented dependency — 37
uniqueness — 23
unit economics — 33
unit economics: cost per thing — 33
unit test — 19, 27
unknown member — 19, 20
unpivot — 18
unplanned work — 40
upcasting — 36
updated_at — 13
upsert — 20

V

vacuum — 7, 10
valid_from — 20
valid_to — 20
validity — 23
vector database — 12
vector databases — 12
vectorization — 22
vendor lock-in — 5
verification — 37, 38
verify by rebuilding — 38
version note and the alternatives — 24
versioning — 17
versioning three things — 27
virtual warehouse — 8, 33
volume — 25
volume floor — 23

W

warehouse versus lake versus lakehouse — 3
watermark — 2, 13, 20, 29
webhooks — 16
what "production-grade" means here — 1
what a contract contains — 17
what a data deploy actually is — 27
what a data engineer should and should not own — 32
what a feature is — 32
what a lakehouse still does not give you — 10
what a slot guarantees, and what that costs — 14
what a table format actually adds — 10
what a test costs — 23
what an event is — 36
what an ml team actually needs from you — 32
what analytics needs instead — 6
what architecture actually decides — 3
what belongs in the orchestrator — 24
what cap actually says — 4
what cdc solves that batch cannot — 14
what changed, and why it was economics — 3
what data engineering is not — 1
what dbt actually is — 19
what duckdb demonstrates faithfully — 8
what duckdb does not show you — 8
what each mode permits — 17
what each one actually is — 22
what happens — 15
what happens to a bad row — 23
what is in a commit — 10
what is on disk — 10
what it buys, quantified — 8
what it costs — 9, 35, 36
what it costs and how long it takes — 38
what it does not — 8
what it does not add — 10
what it is not — 17
what kafka actually is — 15
what latency is actually required — 29
what makes a data team good to work on — 40
what makes a warehouse different — 8
what makes it someone else's — 38
what normalization optimizes — 6
what pages, and what does not — 26
what reproducibility buys — 28
what staff actually is — 40
what storage decisions you actually own — 2
what streaming actually costs — 3
what the job actually is — 1, 40
what the layers are actually for — 34
what the layers buy: replay — 34
what the log actually contains — 14
what the on-call engineer may decide alone — 26
what the overlap costs — 37
what the pattern costs — 34
what the quality register caught — 38
what they are actually screening for — 39
what this book uses, and why — 5
what went wrong, honestly — 38
what would reverse this — 3
what you can safely leave out — 5
what you have built — 38
what you need to know about a source system — 2
what you would do differently — 38
when contracts are not worth it — 17
when cron stops working — 24
when governance is imposed — 30
when it is right — 35
when not to — 36
when not to use cdc — 14
when not to use it — 34
when streaming is actually right — 29
when you cannot fix it tonight — 26
when you find something bad — 31
when you need none of this — 5
where a credential comes from — 28
where a test lives — 23
where etl is still right — 3
where the boundaries actually fall — 34
where the discipline came from — 1
where the money actually goes — 8
where the money actually is — 33
where the schema lives — 11
which errors to retry — 16
whose test is it? — 23
why a lifecycle and not a tool list — 2
why a platform ends up with six databases — 12
why bad data is worse than no data — 23
why it happens — 9
why it has a bad reputation — 30
why keep raw when you have cleaned data — 9
why model at all — 6
why select is both right and wrong here — 13
why this is a data engineering problem — 12
why this is hard, and it is not the code — 37
wide transformation — 21
wide-column store — 12
wide-column stores — 12
window function — 18
window functions — 18
windowing — 29
worker — 24
workload identity — 28
workspace — 28
write-ahead log — 14
writing an as-of join — 32

X

xcom — 24
xcom, and what not to put in it — 24

Y

you are a guest — 7

Z

z-ordering — 10
zero-copy — 22
zone map — 8
zstd — 11

1336 index entries across 40 chapters.