Glossary

Every term declared in a chapter's key_terms appears here, with the chapter that introduces it. scripts/validate.py checks the parity, so a term added to a chapter and not defined here is reported.

Definitions are deliberately short and deliberately opinionated. Where a term is contested — "data mesh," "exactly-once," "real-time" — the entry says so rather than pretending consensus.

A

ABAC (ch30) — attribute-based access control. Permissions computed from attributes of the user, the resource, and the context rather than from a named role. More expressive than RBAC and much harder to audit.

acceptance criterion (ch38) — the single condition that decides whether a project is done. This book's is that gold reconciles to source to the cent, with every difference explained by a documented, tested rule.

accumulating snapshot (ch6) — a fact table with one row per process instance and several date columns filled in as the process advances. Useful for well-defined milestones, awkward because the rows are updated rather than appended.

accuracy (ch23) — whether a value matches reality. The quality dimension assertions cannot check alone, because the platform has no independent access to reality; a reconciliation is the closest substitute.

acks (ch15) — a Kafka producer setting for how many replicas must acknowledge a write. acks=all is the durable choice; the others trade durability for latency invisibly until a broker fails.

action item (ch26) — the output of a postmortem that has an owner and a date. Without both it is a sentiment.

actionability (ch25) — whether an alert tells its recipient what to do. What distinguishes an alert from a notification, and whose absence produces alert fatigue.

adaptive query execution (ch21) — Spark's runtime re-planning from actual statistics: coalescing partitions, switching join strategies, splitting skewed ones. Handles many skew problems and is not a substitute for understanding them.

additive measure (ch6) — a measure that can be summed across every dimension. Revenue is additive; a ratio is not, and summing a non-additive measure is a common modelling error.

after image (ch14) — the state of a row after a change, in CDC. With the before image, what makes a CDC stream more informative than a snapshot.

aggregate (ch36) — in event sourcing, the entity events are about and whose version they are scoped to. If you cannot identify yours, event sourcing is the wrong pattern.

alert fatigue (ch25) — alerts dismissed in bulk because most need no action. Kestrel's was 412 alerts a quarter, 89% requiring none.

allowed lateness (ch29) — how long after the watermark passes a window a processor still accepts events for it. A business decision expressed as configuration.

analytics engineering (ch1) — transforming data in the warehouse with software engineering discipline. Largest job market of the specializations, closest to the business, shallowest technically.

anonymization (ch31) — rendering data no longer relating to an identifiable person, placing it outside the regulations. The bar is far higher than removing names, and the claim is usually false.

Apache Arrow (ch22) — an in-memory columnar format letting engines exchange data without serializing. Why pandas, Polars, and DuckDB can hand data to one another cheaply.

append-only (ch34) — only ever added to, never updated or deleted from. Bronze's requirement, and what makes replay possible.

apply (ch28) — in Terraform, the step that makes planned changes real. The one to gate, review, and log.

approximate nearest neighbour (ch12) — finding close vectors without checking every one. Trades recall for latency in a way that must be measured.

AQE (ch21) — see adaptive query execution.

architecture decision record (ch3) — a short record of a decision, its context, alternatives, and consequences. Its value is entirely in being written before the decision is executed.

artifact (ch27) — a build output that can be stored, versioned, and deployed. In a data platform the important one is dbt's manifest.json, because it makes "what changed" computable.

as-of join (ch32) — a join taking, for each event, the latest matching row at or before its timestamp. What makes training data point-in-time correct.

asset (ch24) — in asset-based orchestration, a declared thing that exists rather than a task that runs. Shifts a DAG from "do these steps" to "make these things true."

at-least-once (ch4) — a message may be delivered more than once and will not be lost. The right default, given idempotent consumers.

at-most-once (ch29) — a message may be lost and will not be duplicated. Almost never what you want for data.

atomic rename (ch10) — making a file visible at its final path in one step. Object stores lack it, which is why table formats exist.

auto-suspend (ch33) — a warehouse setting that stops billing when idle. Disabling it was Kestrel's single largest waste: $5,040 a month for a resource used three hours a day.

autonomy (ch35) — a domain's ability to build, deploy, and change its products without coordinating. What data mesh requires, and what a fan-out analysis measures.

Avro (ch11) — a row-oriented binary format with a schema, common in Kafka. Good for streams, wrong for analytical scans.

B

B-tree index (ch7) — the default relational index. Excellent for point lookups and ranges, useless to a query that does not use it, and never free — every index slows writes.

backfill (ch13) — recomputing history, usually after a bug or a new model. What makes idempotency non-negotiable.

backfill window (ch26) — how far back a pipeline can be re-run, bounded by source retention. Must cover the age of the data you might reconstruct, not your detection time.

backflow (ch34) — a lower layer reading a higher one; silver reading gold. It does not fail, which is the problem: it silently destroys the guarantee that the graph computes from raw.

backpressure (ch4) — a consumer signalling it cannot keep up so the producer slows. The alternative to unbounded buffering, which fails later and worse.

backward compatibility (ch17) — a new producer's output can still be read by an old consumer. Add optional fields; never remove or narrow.

batch (ch29) — processing a bounded set at once. The default, and correct far more often than the streaming literature implies.

batch ingestion (ch13) — loading on a schedule rather than continuously. Simpler, easier to backfill, and sufficient for most requirements.

batch processing (ch3) — see batch.

batch scoring (ch32) — running a model over a set of entities on a schedule. The mode that does not need a feature store.

before image (ch14) — the state of a row before a change. Required to compute a delta, and not emitted by every CDC configuration.

behavioral interview (ch39) — the round asking how you have worked. Mostly probing whether you can be wrong out loud, and whether you would be pleasant on call.

big bang migration (ch37) — cutting over at a single moment. Defensible only when running both is impossible.

binlog (ch14) — MySQL's transaction log, and what most CDC tools read. Postgres' equivalent is the write-ahead log.

bit packing (ch8) — storing integers in the minimum bits their range needs. One of the encodings that makes columnar formats small.

bitmap heap scan (ch7) — Postgres collecting matching row locations from an index into a bitmap, then reading the heap in physical order. Appears when an index helps and is not selective enough for a plain index scan.

blameless postmortem (ch26) — an incident review treating human error as a symptom of system design. Not politeness: the only regime in which people report what actually happened.

blast radius (ch28) — how much a change or failure can affect. In infrastructure, why state files are separated; in a data graph, the models a corruption invalidates.

block compression (ch11) — compressing groups of values together rather than individually. Far better ratios, because it can exploit repetition.

bloat (ch7) — dead tuples left by Postgres updates and deletes, occupying space until vacuumed. Grows silently and degrades scans.

blue-green (ch27) — running two environments and switching between them. Adapts poorly to data, where the "traffic" is a table other things reference.

bottleneck (ch35) — the constraint limiting throughput. Data mesh's premise is that a central team has become one — measurable, and usually unmeasured.

bounded context (ch35) — from domain-driven design, a boundary within which a model and its vocabulary are consistent. Proposed data domains that do not correspond to one are directories.

breaking change (ch17) — a change invalidating a consumer's assumptions. Removing a column, narrowing a type, or changing a value's meaning — the last does not look like a change.

bridge table (ch6) — a table resolving a many-to-many between a fact and a dimension. Necessary and awkward: aggregation is ambiguous unless weights are stored.

broadcast join (ch21) — sending a small table to every executor to avoid a shuffle. The most effective Spark optimization when it applies, and a memory hazard when the table is not small.

broker (ch15) — a Kafka server, holding partitions and serving producers and consumers.

bronze layer (ch34) — the layer holding exactly what the source sent: append-only, untyped, keeping what you do not want. What makes "it arrived that way" provable.

bucket (ch9) — a fixed number of files a table is hashed into, so joins on that column avoid a shuffle. Powerful, inflexible, expensive to change.

budget (ch28) — a spend threshold with an alert; in reliability, the unreliability an SLO permits.

build cache (ch27) — reusing a step's output when its inputs have not changed. The difference between a two-minute and a forty-minute CI run.

burn rate (ch26) — how fast an error budget is consumed, over a short recent window against the rate the SLO permits. Measured over the full window it collapses to "budget spent" and the thresholds cannot fire.

burnout (ch40) — the symptom. The causes are unplanned work above half, custody without authority, no visible outcome, and a ceiling — with different remedies.

bus matrix (ch6) — Kimball's grid of business processes against conformed dimensions. Makes shared dimensions obvious before you build them twice.

business metadata (ch30) — what a dataset means, as opposed to how it is stored. The half of a catalog people actually read.

business rule (ch23) — a decision about meaning that somebody owns. Three of Kestrel's four reconciliation rules belonged to finance, and all three had been decided silently in SQL.

C

canary (ch25) — a deployment that exposes a change to a small share of traffic first. Adapts poorly to data, where the equivalent is shadow running and diffing outputs rather than watching for errors.

CAP theorem (ch4) — under a network partition, a distributed system must choose between consistency and availability. Frequently invoked and rarely the actual constraint; the useful version is that the choice is per-operation, not per-system.

capstone (ch38) — the assembled platform, judged by reconciliation rather than by whether it runs.

cardinality (ch25) — the number of distinct values a column or metric label takes. The property that decides whether a time-series label is fine or catastrophic: customer_id at 1.9M values produces 9.12 billion series.

catalog (ch10) — in a table format, the service mapping a table name to its current metadata pointer. In governance (ch30), the searchable record of what data exists and what it means.

catalyst (ch21) — Spark's query optimizer. Rewrites logical plans, and the reason explain() output looks nothing like the code you wrote.

catchup (ch24) — Airflow's behaviour of running every missed interval when a DAG is unpaused. Extremely useful for backfills and a common cause of a thousand accidental runs.

CCPA (ch31) — the California Consumer Privacy Act. Different structure from GDPR, the same six technical capabilities, a 45-day response window.

certification (ch30) — marking a dataset as trusted, by a named owner, with a date. The status tier that makes a catalog's search useful rather than exhaustive.

change data capture (ch14) — reading a database's transaction log to produce a stream of row changes. Gives update history and low source load, at the cost of operational complexity and a replication slot that can fill the source's disk.

change detection (ch13) — determining which rows have changed since the last load. Usually by updated_at, which lies in four distinct ways.

change stream (ch12) — MongoDB's CDC mechanism. The same idea as a binlog reader, different vocabulary.

chargeback (ch33) — moving cloud spend onto the consuming team's budget. Creates real incentives, and also real incentives to argue about attribution; showback first.

chasm trap (ch6) — a dimensional modelling hazard where two many-to-one relationships from a shared dimension produce a fan-out on join. Detected by row counts, not by inspection.

check strategy (ch20) — how an incremental model decides which rows are new: a timestamp, a high-water mark, a hash, or a CDC log. Each has a different failure mode.

checkpoint (ch21) — writing intermediate state durably so a job can resume. In streaming, the mechanism behind exactly-once processing within a framework's boundary.

chunking (ch13) — reading or writing data in bounded pieces rather than all at once. The difference between a load that works at 10x volume and one that does not.

circuit breaker (ch16) — stopping requests to a failing dependency for a period rather than retrying into it. Protects the dependency as much as the caller.

classification (ch30) — assigning a sensitivity tier to data. A legal determination that engineering makes operable, not one engineering makes.

clear and rerun (ch24) — Airflow's operation to re-execute a task and its downstream. Safe only if the tasks are idempotent, which is why Chapter 20 comes first.

clickstream (ch1) — the event stream of user activity. Kestrel's is 14,000,000 events/day, 11.48 GB/day of JSON, and the largest thing in the platform.

clock skew (ch4) — the difference between machines' clocks. Small, unavoidable, and the reason event-time processing cannot rely on a producer's timestamp being ordered.

clustering key (ch8) — the column order a table's data is physically sorted by. Determines which predicates can skip data; a storage decision with query consequences.

coalesce (ch21) — reducing partition count without a full shuffle. Cheap, and it can produce very uneven partitions, which is sometimes what you wanted and sometimes not.

column chunk (ch9) — in Parquet, one column's data within one row group. The unit that statistics describe and that predicate pushdown skips.

column mapping (ch10) — a table format feature letting a column be renamed without rewriting data, by mapping names to stable field IDs.

column-level lineage (ch30) — tracking which source columns feed which output columns. Expensive to produce, and it answers impact-analysis questions table-level lineage cannot.

columnar (ch22) — see columnar storage.

columnar format (ch11) — a file format storing values column by column: Parquet, ORC. Compresses far better, and lets a query read only the columns it needs.

columnar storage (ch8) — the same idea at the storage-engine level. The single largest performance difference between an analytical and a transactional system.

command (ch36) — a request for something to happen, which can be rejected. Distinct from an event, which has already happened; imperative event names mean you have a command queue.

commit (ch10) — in a table format, atomically publishing a new table version. The operation object stores cannot do natively and that table formats implement.

communications lead (ch26) — the incident role that talks to everyone not fixing the problem. Separating it from the incident commander is what stops the person debugging from writing status updates.

compaction (ch9) — rewriting many small files into fewer large ones. In Kafka (ch36) a different thing entirely: retaining the latest record per key, which destroys an event-sourced log.

completeness (ch23) — whether all the expected data arrived. The quality dimension a row count checks, and the one most likely to fail silently.

compression ratio (ch11) — uncompressed size over compressed size. Kestrel's clickstream is 12.3x from JSON to Parquet, and most of that is the format rather than the codec.

concept drift (ch32) — the relationship between features and the target changing over time. A modelling problem rather than a data engineering one, and frequently blamed for what is actually skew.

concurrency (ch24) — how many tasks may run at once, at the DAG, pool, or cluster level. The setting that turns a resource problem into a queue.

conformed dimension (ch6) — a dimension used consistently across multiple fact tables. What makes cross-process analysis possible, and what silver exists to produce.

connector offset (ch14) — a CDC connector's position in the source log. Losing it means a full re-snapshot; advancing it wrongly means silent data loss.

consent (ch31) — permission to process data for a purpose. Temporal, per-purpose, revocable, and one of several lawful bases — not a boolean.

consistency (ch23) — whether related values agree: across systems, across time, or across two computations of the same metric.

consistent hashing (ch4) — assigning keys to nodes so that adding or removing a node moves as few keys as possible. The mechanism behind partitioned stores that can be resized.

consumer (ch15) — a process reading from a topic. Its offset, not the broker, records what it has seen.

consumer group (ch15) — a set of consumers sharing a topic's partitions, each partition assigned to exactly one member. The unit of parallelism and of rebalancing.

consumer lag (ch15) — how far behind the log's head a consumer is. The single most important streaming metric, and the one that reveals a stalled consumer before its output does.

consumer rebalance (ch4) — reassigning partitions when group membership changes. Necessary, and it briefly stops processing and can replay from a committed offset.

consumer-driven contract (ch17) — a contract asserted by the consumer against the producer, so the producer's tests fail when they break someone. Inverts the usual direction, and is what makes a contract enforceable.

continuous deployment (ch27) — automatically releasing every change that passes CI. In data, the question is what "release" means when the artifact is a table.

continuous integration (ch27) — running the build and tests on every change. For a data platform, building the models against a sample is the version that catches most defects.

contract test (ch16) — a test asserting that an external API still behaves as assumed. The cheapest protection against a vendor changing something quietly.

controller (ch31) — in privacy law, the party that decides why data is processed. Determines who is accountable; a legal determination worth knowing the word for.

Conway's law (ch35) — organizations design systems that mirror their communication structure. Data mesh's foundation — and the inverse maneuver, changing the organization to get the architecture, is a reorganization and should be argued as one.

copy-on-write (ch10) — a table format's update strategy that rewrites whole files on change. Simple reads, expensive writes; the alternative is merge-on-read.

correlated subquery (ch18) — a subquery referencing the outer query's row. Readable and frequently slow; most can be rewritten as a join or a window function.

cost attribution (ch33) — assigning spend to teams and pipelines. 36.4% of Kestrel's bill had no owner, and every dollar of it was waste.

cost per order (ch33) — a unit cost. $0.1526 for Kestrel's platform and $0.0864 after optimization — and the pipeline-only figure of $0.0181 is a different number, and a flattering one.

cost per pipeline (ch33) — spend attributed to one DAG. What turns "compute is up 69%" into a two-hour investigation.

cost per query (ch33) — the estimate that should appear in code review, where it can still change a decision.

coupling (ch3) — how much one component's change forces another's. The dimension along which architecture decisions are actually made.

covering index (ch7) — an index containing every column a query needs, so the table is never read. Fast, and it duplicates data and slows writes.

CPRA (ch31) — the California Privacy Rights Act, which amended CCPA. Evidence for §31.1's warning that this material dates.

CQRS (ch36) — command query responsibility segregation: separate models for writing and reading. In event sourcing it arrives as a consequence rather than as a pattern to adopt.

credit (ch8) — Snowflake's billing unit. A Medium warehouse consumes 4 an hour, at $2.00 each on this book's frozen rate card.

crypto-shredding (ch31) — encrypting each subject's data with a per-subject key and deleting the key to render it unreadable. The best available answer to deleting from an immutable store, and a substantial key-management commitment.

CSV (ch11) — comma-separated values. Ubiquitous, schemaless, ambiguous about types and quoting, and the format your most important vendor will send you.

CTE (ch18) — a common table expression, WITH ... AS. Readability, and in some engines an optimization fence — check yours before relying on it either way.

cursor pagination (ch16) — paginating by an opaque token rather than an offset. Correct under concurrent writes, which offset pagination is not.

cutover (ch37) — the moment consumers switch from the legacy system to the new one. Gated by a readiness rule rather than by a feeling, and preceded by a rehearsed rollback.

backward compatibility (ch17) — a new producer's output can still be read by an old consumer. Achieved by adding optional fields and never removing or narrowing existing ones.

D

DAG (ch24) — directed acyclic graph. The shape of a pipeline's dependencies, and in Airflow the unit of scheduling.

data architecture (ch3) — the set of decisions about where data lives, how it moves, and who may change it. Mostly a record of trade-offs, and worth writing down for that reason.

data catalog (ch5) — the searchable record of what data exists, what it means, and who owns it. Worth having when it answers questions people currently ask a person.

data contract (ch17) — an agreement about a dataset's schema, semantics, and guarantees, enforced by tests rather than by goodwill.

data docs (ch23) — Great Expectations' generated documentation of an expectation suite and its results. Useful because it is a by-product rather than a task.

data downtime (ch23) — the period during which data is missing, late, or wrong. Distinct from system downtime, and usually far longer because nothing alerts on it.

data drift (ch32) — a change in the distribution of a model's inputs. Detected by comparing feature distributions in training and serving, which is the highest-value ML monitor.

data engineering (ch1) — building and operating the systems that move and shape data so somebody can rely on the result. Not machine learning, not analysis, and not plumbing.

data engineering lifecycle (ch2) — generation, ingestion, storage, transformation, and serving, with governance across all five. A map for locating a problem, not a methodology.

data impact (ch26) — the required field in an incident record stating which data is wrong, for which period, and who consumed it. What turns an outage report into a data incident report.

data interval (ch24) — in Airflow, the window of data a run is responsible for, as distinct from when it executes. Confusing them is the most common Airflow bug.

data lake (ch1) — files in an object store, queried in place. Cheap, flexible, and without a table format it has no atomicity, no schema evolution, and no row-level delete.

data lakehouse (ch3) — a lake with a table format on top, giving warehouse-like guarantees over open files. The architecture most new platforms land on.

data leakage (ch32) — training on information that did not exist at prediction time. Produces a model that scores excellently offline and badly in production, immediately.

data mesh (ch35) — domain-oriented ownership, data as a product, a self-serve platform, and federated computational governance. A real answer to a real coordination problem, and the most misapplied idea in the field.

data minimization (ch31) — collecting and keeping only what you need. The cheapest privacy control and the most under-used, because dropping a column feels like losing an option.

data pipeline (ch1) — a sequence of steps that moves and transforms data. The unit most of this book is about.

data product (ch2) — a dataset treated as a product: a known consumer, a contract, an SLO, documentation, and an owner who can act.

data product owner (ch35) — the person accountable for a data product's quality and roadmap. In a mesh, a domain role; without a mesh, still a useful one.

data quality (ch1) — whether data is fit for the decisions made on it. Measured by assertions, not by inspection.

data residency (ch31) — a requirement that data remain in a jurisdiction. Constrains storage, processing, and — the part that gets missed — logs, metrics, and error payloads.

data subject (ch31) — the person data is about. The unit privacy obligations operate on, and the access pattern most platforms are bad at.

data swamp (ch9) — a lake with no catalog, no conventions, and no owners. The predicted outcome of a lake adopted without governance, and a common one.

data transfer (ch33) — moving bytes between regions, zones, or out to the internet. Priced irrationally, easy to incur accidentally, and 5.6% of Kestrel's bill.

data warehouse (ch1) — a database optimized for analytical queries over modelled data. Still the right default for most organizations.

DataFrame API (ch21) — Spark's typed, optimizer-visible API, as opposed to RDDs. Almost always the right choice, because Catalyst can see what you meant.

DataOps (ch2) — applying DevOps practices to data: version control, CI, monitoring, and small reversible changes. Useful as a checklist, weak as an identity.

dataset (ch24) — in Airflow, a declared data object that can trigger a DAG when updated. The bridge between task-based and asset-based scheduling.

dbt package (ch19) — a reusable set of dbt models, macros, and tests. dbt_utils is the one nearly every project ends up depending on.

dead letter (ch23) — a record that failed processing and was set aside rather than dropped. Only useful if somebody drains the queue.

dead letter queue (ch15) — the streaming version. A queue nobody drains is a queue that grows until somebody truncates it.

Debezium (ch14) — the most widely used open-source CDC platform, reading database logs and producing Kafka records. Also the standard way to implement the outbox pattern.

declarative (ch28) — describing the desired end state rather than the steps to reach it. Terraform's model, and the reason a plan can be reviewed before it is applied.

decommissioning (ch37) — removing a migrated system: the job, the code, the credentials, and the infrastructure. Deleted, not disabled — a disabled job gets re-enabled.

deduplication (ch18) — reducing multiple rows for one entity to one. Requires a deterministic tie-break, or the result differs between runs and no uniqueness test detects it.

defer (ch19) — dbt's mechanism for resolving unbuilt models against another environment, so CI can build only what changed.

deferrable operator (ch24) — an Airflow operator that releases its worker slot while waiting. The fix for sensors deadlocking a pool.

definition of done (ch38) — the condition under which work is finished. For a pipeline it is not "it runs"; it is "it reconciles."

degenerate dimension (ch6) — a dimension attribute stored on the fact table because it has no other attributes, such as an order number. Correct, and it looks like a mistake.

degraded mode (ch26) — running with reduced function rather than stopping. Worth designing in advance, because the alternative is deciding during an incident.

delete+insert (ch20) — an incremental strategy that deletes a partition and re-inserts it. The simplest way to be idempotent, and it needs a well-chosen partition key.

deletion vector (ch10) — a file recording which rows in a data file are deleted, so a delete does not rewrite the file. What makes row-level deletes affordable and privacy requests tractable.

delta encoding (ch8) — storing differences between consecutive values rather than the values. Very effective on sorted or slowly-changing columns.

denormalization (ch12) — duplicating data to avoid joins. Right in a document store and in a serving layer; a source of update anomalies everywhere else.

DENSE_RANK (ch18) — a window function ranking without gaps after ties. Distinct from RANK and ROW_NUMBER, and the distinction is a common interview question because it is a common bug.

deprecation (ch17) — announcing that something will be removed, with a date, while it still works. The half of a breaking change that makes it survivable.

depreciation (ch40) — the loss of value in a skill over time. Products depreciate; properties do not.

dictionary encoding (ch8) — replacing repeated values with integer references to a dictionary. The encoding that makes low-cardinality string columns nearly free.

dictionary page (ch11) — where Parquet stores that dictionary, per column chunk.

differential privacy (ch31) — adding calibrated noise so that an individual's presence cannot be inferred, with a mathematical guarantee that survives composition. Rigorous, and the noise is large enough to matter for business analytics.

digest pinning (ch28) — referencing a container image by content hash rather than by tag. The difference between a reproducible deploy and a hopeful one.

dimension table (ch6) — a table of descriptive attributes joined to facts. Wide, small, and where the business's vocabulary lives.

dimensional modeling (ch6) — organizing a warehouse into facts and dimensions. Thirty years old and still the default for good reasons.

discoverability (ch30) — whether somebody can find the right dataset without asking a person. The catalog's actual job.

distribution (ch25) — the shape of a metric's values, as opposed to its average. Where the interesting failures live, and what a p99 exists to expose.

distribution test (ch23) — an assertion about a column's distribution rather than its individual values. Catches drift that row-level checks cannot.

Docker (ch28) — containerization. In a data platform, mostly the mechanism for making a job's environment reproducible.

docker compose (ch5) — running a multi-container local environment from one file. How this book's sandbox runs Postgres, MinIO, and Kafka on a laptop.

document store (ch12) — a database storing schemaless documents, typically JSON. Good for varied shapes and heterogeneous access; bad for analytical aggregation.

domain (ch35) — in a mesh, a business area that owns its data. Real only if it can build and change its products without coordinating.

domain-oriented ownership (ch35) — the mesh principle that the team producing data owns the data product built from it. The only one of the four that requires decentralization.

downsampling (ch12) — reducing the resolution of time-series data as it ages. The standard way to keep a metrics store from growing without bound.

drift (ch28) — infrastructure differing from its declared state, usually from a console change during an incident. Detected by a scheduled plan, not by hoping.

driver (ch21) — the Spark process that plans the job and coordinates executors. Also where collecting a large result will run you out of memory.

DSAR (ch31) — a data subject access request: produce everything you hold about a person. Harder than deletion, because "everything" is a judgment about scope.

dual write (ch36) — writing to a database and then publishing an event, without a transaction spanning both. Loses events silently; the fix is a transactional outbox.

DuckDB (ch5) — an in-process analytical database. Fast, dependency-free, and the reason this book can teach warehouse concepts without a cloud account.

duration (ch25) — how long a job took. Monitored with an absolute threshold in most platforms, which cannot detect a job doing ten times the work in three times the time.

dynamic allocation (ch21) — Spark adding and removing executors based on demand. Keeps wall-clock stable and decouples cost from latency, which removes a signal (ch33).

dynamic data masking (ch31) — transforming a column's value at query time based on the querying role. Attached to the object, auditable in one place, and fails closed.

dynamic task mapping (ch24) — Airflow generating tasks at runtime from a list. Useful, and it makes the DAG's shape depend on data, which complicates monitoring.

E

eager evaluation (ch22) — computing each operation as it is written. pandas' model; simple to debug and it materializes intermediates you did not need.

egress (ch33) — data leaving a cloud region or provider. $0.09/GB on this book's rate card, and the least rational line on any bill.

elasticity (ch3) — the ability to add and remove capacity on demand. What the cloud sells, and it only saves money if you actually release the capacity.

ELT (ch1) — extract, load, transform: land raw data first and transform in the warehouse. The default today, and wrong when you legally may not land the raw data.

embedding (ch12) — a vector representation of an item, used for similarity search. A model output, so it has a version as well as a value.

enabling team (ch35) — from Team Topologies, a team that helps others acquire a capability rather than doing the work for them. What a platform team becomes in a mesh.

entity (ch32) — the thing a feature is about: a customer, an order, a product, or a pair. Its key is a join key, and Chapter 6's discipline applies.

entity key (ch32) — the identifier a feature is stored and looked up by.

environment (ch27) — a separate place to run: dev, staging, production. In data, the hard part is that staging cannot have production's data (ch31).

ephemeral model (ch19) — a dbt model compiled into its consumers as a CTE rather than materialized. Cheap, and it makes debugging harder because it does not exist.

erasure (ch31) — deleting everything about a person, within a statutory deadline. Requires finding them first, in every identifier space.

error budget (ch25) — the amount of unreliability an SLO permits. What turns a reliability argument into an arithmetic one.

escalation (ch26) — handing an incident to someone with more context or more authority. Should have a stated trigger, or it happens too late.

ETL (ch1) — extract, transform, load: transform before landing. Right when you cannot store the raw data, and the strongest argument for it is legal rather than technical.

event (ch36) — an immutable, past-tense statement that something happened, identified and versioned within its aggregate.

event carried state transfer (ch36) — an event carrying the new state of an entity, so consumers need not call back. What CDC produces, and what most "event-driven" data platforms actually run on.

event envelope (ch14) — the metadata wrapping a CDC record: source, operation, timestamps, and position. Distinct from the payload, and where the useful debugging information lives.

event notification (ch36) — an event carrying only "X changed" and an id. Cheap and loosely coupled, and it creates a synchronous dependency back on the producer.

event sourcing (ch36) — treating the event log as the system of record, with every table derived. A large commitment whose payoff is retroactive projections and debugging.

event store (ch36) — the durable, append-only home of events.

event time (ch4) — when something happened, as opposed to when it was processed. The distinction the whole of stream processing is organized around.

event-driven architecture (ch36) — designing around events rather than tables. Covers three different things with different costs, and conflating them is the main source of over-engineering.

eventual consistency (ch4) — replicas converge given no new writes. Fine for many things and surprising when a read-after-write returns the old value.

exactly-once (ch4) — a guarantee that a message is processed exactly once. Real within a framework's boundary and not across a boundary; the practical version is at-least-once plus idempotent consumers.

executor (ch21) — a Spark process that runs tasks and holds cached data.

expand-contract migration (ch17) — add the new thing, migrate consumers, then remove the old one. The pattern that makes a breaking change non-breaking, at the cost of a period with both.

expectation suite (ch23) — a named set of assertions in Great Expectations.

exponential backoff (ch16) — increasing the wait between retries. With jitter, the difference between recovering from an outage and prolonging it.

exposure (ch19) — a dbt declaration of a downstream consumer — a dashboard, an ML model, an export. Under-used, and it directly answers "if this breaks, who notices?"

extract manifest (ch13) — a record of what a load extracted: files, row counts, and the high-water mark. Makes a load auditable and re-runnable.

F

fact table (ch6) — a table of measurements at a declared grain, joined to dimensions. Long, narrow, and the thing everything else exists to describe.

factless fact table (ch6) — a fact table with no measures, recording that an event occurred. Attendance, coverage, and eligibility are the usual cases.

fan trap (ch6) — a join that multiplies rows because a one-to-many relationship is traversed before an aggregation. Produces a number that is too large and looks plausible.

feature (ch32) — a value about an entity, at a moment. The third part is the one that gets dropped.

feature age (ch32) — the gap between when a feature value was computed and when it is used to predict. Almost never measured, and a 132x mismatch between training and serving is normal.

feature freshness (ch32) — how recent the served feature is. Must be compared against training, not minimized.

feature store (ch32) — an offline store for training and an online store for serving, sharing one feature definition. Worth it only with online inference, duplicated computation, and more than one consumer.

feature versioning (ch32) — tracking a feature definition's changes, so a model can be pinned to the one it was trained on. A materially changed definition is a new feature with a new name.

federated computational governance (ch35) — global standards decided by a federation and enforced by automation. The best idea in the mesh, and the computational half works without any decentralization.

file compaction (ch33) — merging small files into larger ones. Improves scan performance and request costs, and its cost justification is usually much weaker than its latency justification.

FinOps (ch33) — the practice of managing cloud spend as an engineering concern. Useful as a checklist; be skeptical of its maturity models.

fixed and variable cost (ch33) — the split that makes a unit cost predictive. Kestrel is ~64% variable, so doubling volume raises the bill 64% and lowers cost per order 18%.

FinOps — see above.

fixture (ch27) — a small, fixed dataset used for testing. The thing that makes a CI run fast enough to be run on every change.

fold (ch36) — reducing a sequence of events to a state. state = fold(reducer, events) is event sourcing's central identity.

footer (ch9) — the end of a Parquet file, holding the schema and the row-group statistics. Read first, which is why a reader can skip most of a file.

format-preserving encryption (ch31) — encryption whose output has the same shape as its input. Useful when a downstream system validates the format of a value it does not need to read.

forward compatibility (ch17) — an old producer's output can still be read by a new consumer. The mirror of backward compatibility, and the one people forget to test.

forward fix (ch27) — resolving a bad deploy by shipping a correction rather than rolling back. Frequently the only option in data, because the previous state has been overwritten.

frame clause (ch18) — the ROWS/RANGE/GROUPS part of a window function. RANGE is the default, it includes peer rows, and the difference matters whenever the ordering column has duplicates.

freshness (ch25) — how old the data is. The first thing to monitor and the one most often expressed as "did the job run" instead.

freshness check (ch23) — an assertion that data is no older than a stated threshold.

full compatibility (ch17) — both backward and forward compatible. The strictest schema-evolution mode and the one that permits the fewest changes.

full load (ch13) — extracting everything, every time. Simple, correct, and it stops scaling at a size you can compute in advance.

full refresh (ch20) — rebuilding a model from scratch rather than incrementally. The default in dbt, and the most common expensive mistake in a mature project.

G

gaps and islands (ch18) — the SQL pattern for finding runs of consecutive values, usually by differencing a row number against the value. Sessionization is its most common instance.

GDPR (ch31) — the EU General Data Protection Regulation. Six technical capabilities, a one-month response deadline, and a definition of personal data broader than most engineers assume.

generate (ch2) — the lifecycle stage where data comes into existence, usually in a system you do not control. The only stage where you can prevent a problem rather than handle it.

generic test (ch19) — a reusable dbt test applied by name in YAML: unique, not_null, accepted_values, relationships. Four lines that catch most schema-level defects.

gold layer (ch34) — the layer answering business questions at a documented grain, with named and owned definitions. Disposable: reconstructible from silver.

grain (ch2) — what one row represents. The single most important sentence about a table, and the field most often missing from a catalog.

grain test (ch23) — an assertion that the declared grain holds: a uniqueness check on the grain columns. Nine lines of dbt YAML, and it catches every fan-out.

grant (ch30) — a permission on a database object. Accumulates, never gets narrowed, and 62 identities had one at Kestrel while 23 used it.

Great Expectations (ch23) — a Python data-validation framework. Expressive, with generated documentation, and heavier than a dbt test for the same assertion.

gross merchandise value (ch1) — total sales value before deductions. Kestrel's is $182.0M a year, and it is the denominator for the unit costs in Chapter 33.

gross revenue (ch38) — revenue before refunds are netted, and before the rules that remove test, cancelled, and gift-card rows. Compute Kestrel's in Chapter 38; it is deliberately not printed here.

group coordinator (ch15) — the Kafka broker managing a consumer group's membership and offsets.

GUI ETL (ch37) — a graphical ETL tool such as Informatica or SSIS. The hardest legacy category, because the logic is in a proprietary repository rather than in a file you can read.

gzip (ch11) — a general-purpose compression codec. Good ratios, slow, and not splittable — which matters when a single large file becomes a single task.

H

handoff (ch26) — transferring an incident between responders. Needs a stated protocol, or context is lost precisely when it is expensive.

handover (ch38) — making a platform someone else's to run. Six artifacts, of which the "not done" list is the most valuable and the most often skipped.

hard delete (ch13) — physically removing a row, as opposed to flagging it. Invisible to a timestamp-based incremental load, which is why deletes need CDC or a reconciliation.

hash diff (ch20) — hashing a row's attributes to detect change without comparing every column. The mechanism behind efficient SCD Type 2 loads.

heap (ch7) — the unordered table storage a Postgres index points into. A bitmap heap scan is what happens when the index says "these rows" and the table must be visited.

heartbeat (ch25) — a periodic signal that a process is alive. Detects a stopped scheduler, which is the failure that produces no events at all and therefore no alerts.

high-water mark (ch13) — the maximum value of a monotonic column seen in the last load, used to select new rows. Simple, and it silently misses updates to old rows and any hard delete.

Hive-style partitioning (ch9) — encoding partition values in directory names, year=2026/month=11/day=27. Universally understood and the reason a path is metadata.

hot partition (ch4) — a partition receiving disproportionate traffic because of key skew. High cardinality does not prevent it: 1.9 million customers, one at 8.1%.

HUGEINT (ch22) — DuckDB's 128-bit integer, which SUM(BIGINT) returns. It has no Parquet primitive, so it lands as a DOUBLE and money becomes a float.

I

idempotency (ch1) — running an operation twice produces the same result as running it once. The single most important property in this book.

idempotency key (ch16) — a caller-supplied identifier letting an API deduplicate retried requests.

idempotent consumer (ch36) — a consumer that ignores a duplicate, usually by storing the last version seen per aggregate. What makes the outbox pattern's duplicates harmless.

idempotent producer (ch15) — a Kafka producer that deduplicates its own retries. Prevents reordering within a partition as a side effect.

idempotent rebuild (ch34) — rebuilding a model from its inputs and getting the same result. The property the quarterly rebuild verifies and that three defects in Chapter 38 violated.

idempotent sink (ch29) — a stream destination where writing the same record twice has no additional effect. What makes at-least-once delivery acceptable.

image (ch28) — a container image. Pin it by digest, not by tag.

immutability (ch34) — never changing what has been written. Bronze's rule, event sourcing's rule, and the property that makes history trustworthy.

import (ch28) — bringing an existing resource under Terraform's management. The operation that converts drift into declared state.

in-sync replica (ch15) — a Kafka replica caught up with the leader. acks=all waits for these, and min.insync.replicas decides how many must exist for a write to succeed.

incident commander (ch26) — the person coordinating an incident, who should not also be the person debugging it.

incremental load (ch13) — loading only what has changed. Necessary at scale, and every check strategy for deciding "what changed" has a failure mode.

incremental migration (ch37) — replacing a legacy system one piece at a time, with both running. The strangler fig, and the default unless running both is impossible.

incremental model (ch20) — a dbt model that processes new rows rather than rebuilding. Requires a unique key and an idempotent strategy, or reruns double-count.

incremental snapshot (ch14) — Debezium's ability to snapshot a table in chunks while streaming continues. Removes the long lock that made initial CDC loads painful.

index scan (ch7) — reading an index to find rows. Fast when selective; slower than a sequential scan when it is not, which is why the planner sometimes ignores your index.

individual contributor (ch40) — an engineer without direct reports. A track, not a consolation, and the staff and principal levels are on it.

inference (ch32) — running a model to produce a prediction. Batch or online, and which one decides most of the feature infrastructure.

infrastructure as code (ch28) — declaring infrastructure in version-controlled files. Makes a change reviewable before it is real, which is most of the value.

ingest (ch2) — the lifecycle stage where data enters your systems. The stage with the most failure modes and the least glamour.

ingestion tool (ch5) — Fivetran, Airbyte, Meltano, or code. Buy for standard sources, build for the ones your business depends on.

insert_overwrite (ch20) — an incremental strategy replacing whole partitions. Idempotent by construction, and it requires the partition to align with the load's unit.

integer cents (ch7) — storing money as an integer number of the smallest unit. This book's rule, and validate.py fails the build on a float money column.

integration test (ch27) — a test exercising components together. In data, usually "build the models against a fixture and assert the output."

interoperability standard (ch35) — a global rule every domain follows so their products compose. The "computational" half of federated governance.

inverse Conway maneuver (ch35) — changing the organization to get the architecture you want. Powerful, legitimate, and a reorganization — which is who must approve it.

inverted index (ch12) — a map from term to the documents containing it. The structure behind text search.

is_current (ch20) — the flag marking the active row of an SCD Type 2 dimension. Exactly one per natural key, and asserting that catches every Type 2 bug.

J

jitter (ch16) — randomness added to a retry delay so that clients do not retry in lockstep. Without it, backoff synchronizes a thundering herd.

job (ch21) — in Spark, the work triggered by one action. Divided into stages at shuffle boundaries.

JSON Lines (ch11) — one JSON object per line. Splittable, streamable, verbose, and the format most event data arrives in.

JSONB (ch7) — Postgres' binary JSON type, indexable with GIN. Useful for genuinely variable attributes; a trap when used to avoid modelling.

junk dimension (ch6) — a dimension combining several low-cardinality flags into one table. Keeps the fact table narrow, and it is exactly as inelegant as its name.

K

k-anonymity (ch31) — every combination of quasi-identifiers appears at least k times. A floor, not a target; and a segment extract's k is much worse than its population's.

Kafka (ch15) — a distributed, partitioned, replicated log. The default event backbone, and more operationally involved than its API suggests.

Kafka Connect (ch14) — a framework for moving data into and out of Kafka with configuration rather than code. Where Debezium runs.

Kappa architecture (ch29) — one streaming path, with reprocessing by replaying the log. The response to Lambda's duplicated logic, and its own answer is now mostly obsolete too.

key (ch9) — in an object store, the full path of an object. There are no directories; the slashes are part of the name.

key-value store (ch12) — a database with get and put by key. Fast, simple, and it pushes every access pattern into the key design.

Kubernetes (ch28) — a container orchestrator. In a data platform, mostly a way to run jobs; adopt it because you already have it, not to run a scheduler.

L

l-diversity (ch31) — requiring the sensitive attribute to vary within each equivalence class. k-anonymity's answer to the homogeneity attack, and the first thing anybody asks about after k.

label (ch32) — the target a model is trained to predict. Needs a definition, two timestamps, a delay, and a known source — the same rigor as a feature.

labeling delay (ch32) — the gap between an event and knowing its label. A 90-day churn definition makes your newest training data a quarter old.

LAG (ch18) — a window function returning a previous row's value. With LEAD, the pair that turns a self-join into one pass.

lakehouse (ch10) — see data lakehouse.

Lambda architecture (ch29) — parallel batch and streaming paths reconciled at serving. A careful response to 2011's constraints, and worth reading as history.

late-arriving data (ch13) — records that arrive after the window they belong to. Handled by a lookback window in batch and by watermarks in streaming.

late-arriving dimension (ch20) — a fact arriving before its dimension row exists. Handled by an inferred member or by deferring the fact; both are decisions worth writing down.

latency (ch3) — the delay between a cause and its effect being visible. In data, ask for it as a number in seconds, because "real-time" means four different things.

lateral join (ch18) — a join whose right side can reference the left side's columns. How an as-of join is written portably.

lawful basis (ch31) — the legal ground for processing personal data. Consent is one of several, and a consent check on a flow running on contractual necessity is a bug.

layer (ch28) — in containers, a filesystem diff; ordering them by change frequency is what makes builds fast. In ch34, one of bronze/silver/gold.

layer boundary (ch34) — the interface between two layers, and the place a guarantee is asserted. Free to draw and worthless unenforced.

layer drift (ch34) — a model doing work that belongs to a different layer, acquired one four-line shortcut at a time. The normal end state without enforcement.

layer skipping (ch34) — a layer reading two levels down, such as gold reading bronze. Bypasses every guarantee silver provides, and forces gold to cast.

lazy evaluation (ch21) — building a plan and executing only when a result is required. Lets the optimizer see the whole query, and makes debugging harder because nothing has happened yet.

LEAD (ch18) — a window function returning a following row's value.

leader-follower (ch4) — one replica accepts writes and others copy it. The default replication model, and the reason a failover has a window of possible data loss.

least privilege (ch30) — granting only the access required. Easy to state, and it decays because narrowing a grant has a cost and no visible benefit.

legacy system (ch37) — a system that cannot be safely changed: no tests, no owner, no understanding. A property of your relationship with it, not of its age.

levelling (ch39) — assigning a role to a level. Decided by scope, mostly in the recruiter screen, and under-levelling is far harder to correct after joining.

lifecycle policy (ch9) — a rule that transitions or expires objects by age. The enforcement half of every retention policy that currently has only a period.

lineage (ch25) — the record of which datasets derive from which. Answers impact analysis, and the question people actually ask is "if this breaks, who notices?"

lineage depth (ch34) — how many transformations a model is from raw. A structural measurement that survives relabelling, and a stg_ model at depth 3 is a self-evident contradiction.

link header (ch16) — the HTTP header carrying pagination URLs. Preferable to constructing them yourself, because the server knows its own cursor semantics.

lint (ch27) — automated style and correctness checking. In a data project, SQLFluff plus the project rules that encode your conventions.

liquid clustering (ch10) — Delta's incremental clustering that avoids full rewrites. A newer answer to the problem bucketing solved rigidly.

live coding (ch39) — an interview round writing code while observed. Scored on process, so narrate, and say what you are stuck on.

load-bearing spreadsheet (ch37) — a spreadsheet in a production path, maintained by a person. Not a technical migration: separate the rules from the judgments and migrate only the rules.

log (ch25) — an append-only record of events. In ch36, the source of truth rather than a diagnostic.

log compaction (ch14) — Kafka retaining only the latest record per key. Correct for a state-transfer topic, destructive for an event-sourced one.

log sequence number (ch13) — a monotonic position in a database's write-ahead log. The deterministic tie-break a dedup needs.

logical date (ch24) — Airflow's identifier for the data interval a run covers, distinct from the wall clock. Keying tasks on it is what makes reruns idempotent.

logical decoding (ch14) — Postgres' mechanism for turning WAL records into row-level changes. What Debezium consumes.

logical replication (ch7) — Postgres replication at the row level rather than the block level. The foundation CDC is built on.

long-running transaction (ch7) — a transaction held open, preventing vacuum and growing bloat. A common cause of a Postgres database degrading for no apparent reason.

lookback window (ch20) — reprocessing the last N days on every incremental run, to catch late-arriving data. The cheap approximation of a watermark.

LSN (ch14) — see log sequence number.

LZ4 (ch11) — a fast compression codec with modest ratios. The right default when CPU matters more than bytes.

M

managed service (ch5) — a component the vendor operates. Buy it unless operating it is your differentiator, and remember you are buying their failure modes too.

manifest (ch9) — a file listing the data files in a table version. What makes a listing cheap and a commit atomic.

margin (ch25) — in monitoring, the gap between current performance and the threshold. Publish it rather than the raw value: a deadline met by eight minutes and by three hours look identical otherwise.

mart (ch19) — a set of gold models serving one business area. A packaging convention, not a guarantee.

materiality (ch38) — the size below which a difference is not worth chasing. Reasonable against an external source and not reasonable for a system compared against itself, where the correct difference is zero.

materialization (ch18) — writing a query's result rather than recomputing it; in dbt, the choice of table, view, incremental, or ephemeral. In ch32, copying offline features into the online store.

materialized view (ch8) — a stored, refreshable query result. Fast reads, and a refresh strategy you now own.

medallion architecture (ch34) — bronze, silver, gold. Nearly free, applies almost anywhere, and worth nothing unenforced.

memory multiplier (ch22) — how much RAM an engine needs relative to the data. pandas' is roughly 5–10x and is the reason a "small" file exhausts a laptop.

mentorship (ch40) — improving someone else's work deliberately. One of the four kinds of senior evidence, and the one that leaves no artifact unless you record it.

merge (ch20) — an upsert: update matching rows, insert the rest. dbt's default incremental strategy on engines that support it, and it needs a genuine unique key.

merge-on-read (ch10) — a table format update strategy writing delete vectors and new rows, resolved at read time. Cheap writes, more expensive reads, and compaction to reconcile.

metadata (ch30) — data about data. Technical (types, sizes, lineage) and business (meaning, grain, ownership); the second is the half that gets read.

metric (ch25) — a measured value over time. In ch30, a business quantity with a definition somebody owns.

metric definition (ch30) — the rule computing a metric. Defined once or defined four times, and Kestrel had four definitions of active_customer for twenty-eight months.

micro-batch (ch3) — processing small batches frequently. Most of streaming's benefit with most of batch's simplicity, and usually the right answer to "we need real time."

micro-partition (ch8) — Snowflake's automatically-managed storage unit. Removes manual partitioning and makes the clustering key the lever that remains.

microbatch (ch20) — dbt's incremental strategy processing a bounded time window per batch. Backfills become parallel and reruns become idempotent by construction.

migration (ch27) — a change to a schema or a system. In ch37, replacing a legacy platform: the work most data engineers are actually hired to do.

MinIO (ch5) — an S3-compatible object store that runs locally. Lets this book's boto3 code be unmodified against a laptop.

ML infrastructure (ch40) — the specialization serving data to models in production. Highest ceiling, most exposure to a moving field.

model registry (ch32) — the record of trained models, their versions, and the data they were trained on. Where a content-addressed training snapshot is referenced.

modern data stack (ch5) — the ELT-plus-cloud-warehouse assembly of the early 2020s. A useful shorthand and a phrase that will date.

module (ch28) — a reusable Terraform component. The unit that turns copied HCL into something reviewable.

monitoring (ch25) — observing a running system. For data, freshness, volume, and distributions — not just whether the process exited zero.

multi-stage build (ch28) — a Dockerfile that builds in one image and copies artifacts into a smaller one. Smaller images, fewer packages in production, faster pulls.

multipart upload (ch9) — uploading a large object in parts. Necessary above a size limit, and abandoned parts are billed until a lifecycle rule removes them.

MVCC (ch7) — multi-version concurrency control: readers see a snapshot rather than blocking. Why Postgres readers do not block writers, and why bloat exists.

N

narrow transformation (ch21) — a Spark operation where each output partition depends on one input partition. No shuffle, and therefore cheap.

natural key (ch6) — the business identifier for an entity, as opposed to a surrogate key. What a reconciliation joins on, and what survives a rebuild.

net revenue (ch38) — revenue after refunds are netted. Because refunds settle for up to 90 days, it is not final when the month closes, so the figure needs an as-of date — which is why Chapter 38's is stated as of a specific day, and why it is not printed here.

node selection (ch19) — dbt's --select syntax for choosing models by name, tag, path, or graph relation. state:modified+ is the one that makes CI fast.

non-additive measure (ch6) — a measure that cannot be summed across dimensions, such as a ratio or a balance. Summing one is a common and invisible error.

non-determinism (ch37) — producing a different result from the same inputs. Legacy jobs that depend on scan order cannot be matched, only reconciled against a property.

normalization (ch6) — removing redundancy by splitting tables. Right in an operational database, usually wrong in a warehouse's serving layer.

nullable dtype (ch22) — pandas' newer types that represent missing values without promoting to float. What stops an integer column silently becoming a float.

numeric type (ch7) — Postgres' exact decimal type. Correct for money, slower than integers, and this book prefers integer cents.

O

OAuth (ch16) — a delegated authorization framework. In ingestion, mostly a token you must refresh and must not log.

object dtype (ch22) — pandas' catch-all type, usually holding Python objects. Slow, memory-hungry, and what a string column becomes by default.

object storage (ch9) — S3 and its equivalents: a flat namespace of immutable objects addressed by key. There are no directories; the slashes are part of the name.

observability (ch25) — being able to answer questions about a system from its outputs. For data, the questions are about the data rather than about the process.

observability tool (ch5) — a product that monitors data freshness, volume, and schema. Good, and frequently bought to substitute for practice rather than to extend it.

offer negotiation (ch39) — the conversation at the end. Spend the energy on scope before salary; the cost of a wrong job is a year.

offline store (ch32) — the feature store's historical half, holding all versions for as-of joins. A Type 2 dimension is exactly this.

offset (ch15) — a consumer's position in a partition. Committed by the consumer, which is why processing and committing must be ordered carefully.

offset pagination (ch16) — paginating with LIMIT/OFFSET. Simple, and it skips or repeats rows when the underlying data changes between pages.

OIDC (ch27) — OpenID Connect. In CI, the mechanism for getting short-lived cloud credentials without storing a secret.

OLTP (ch7) — online transaction processing: many small reads and writes. The workload row-oriented storage and B-trees are built for.

on-demand pricing (ch8) — paying per query or per unit of usage with no commitment. Higher unit price, no risk, and the right default until your usage is predictable.

online inference (ch32) — scoring a model on a live request. One of the three conditions that makes a feature store worth it.

online store (ch32) — the feature store's serving half: latest value only, key-value, sub-10ms. Priced by writes, which is the number nobody estimates.

onsite loop (ch39) — the set of interviews after the screen. System design and correctness together are usually more than half the decision.

OpenTelemetry (ch25) — a vendor-neutral standard for traces, metrics, and logs. Worth adopting for the portability rather than for any single feature.

operator (ch24) — in Airflow, a unit of work. Prefer a few well-understood ones to the ecosystem's long tail.

optimistic concurrency (ch10) — attempting a commit and retrying if the table version moved. How table formats let multiple writers proceed without locking.

ORC (ch11) — a columnar format contemporary with Parquet, dominant in the Hive ecosystem. Comparable technically, and Parquet won on adoption.

orchestration (ch1) — deciding what runs, when, and in what order, and what happens when something fails.

orchestrator (ch5) — the tool that does it: Airflow, Dagster, Prefect, or cron until it stops being enough.

ordering guarantee (ch36) — a promise about the order in which records are seen. Needed per-aggregate, rarely globally, and not at all for a commutative projection.

out-of-core (ch22) — processing data larger than memory by streaming it. What DuckDB and Polars do and pandas does not.

output port (ch35) — a data product's documented, versioned interface. Framing eleven ad-hoc extracts as ports collapsed them to three at Kestrel.

ownership (ch17) — who is accountable for a dataset. A team, able to act, verified recently — not the original author.

P

PACELC (ch4) — an extension of CAP: else, when there is no partition, a system trades latency against consistency. The more useful formulation, because the "else" branch is where you live.

page (ch7) — the fixed-size block a database reads and writes. Why a row's width affects scan cost more than intuition suggests.

page token (ch16) — an opaque cursor identifying a position in a result set. Treat it as opaque even when it obviously encodes something.

pagination (ch16) — returning results in pages. Cursor-based where correctness matters; offset-based where it does not.

paging (ch26) — waking a human. Reserve it for something both urgent and actionable, or you have built alert fatigue with consequences.

pandas (ch22) — the default Python DataFrame library. Ubiquitous, eager, memory-hungry, and correct to use for data that fits comfortably.

parallel run (ch37) — running old and new systems simultaneously and comparing. The technique the whole of Chapter 37's verification rests on.

Parquet (ch9) — the standard columnar file format. Compresses well, supports predicate and projection pushdown, and is the default for anything analytical.

partial failure (ch4) — some components failing while others continue. The defining property of a distributed system and the reason idempotency matters.

partition (ch15) — in Kafka, an ordered log within a topic and the unit of parallelism and ordering. In storage (ch4), a physical division of data.

partition alignment (ch20) — arranging a model's partitions to match its incremental unit, so a rerun replaces exactly one partition.

partition assignment (ch15) — mapping partitions to consumers in a group. Changes on rebalance, which is why a consumer can see a replay.

partition by (ch18) — a window function's grouping clause. Determines the frame's scope, and with ORDER BY determines whether peers exist.

partition key (ch4) — the value determining which partition a record goes to. Decides ordering guarantees and skew simultaneously.

partition pruning (ch1) — skipping partitions a predicate cannot match. The highest-value optimization in this book, and a function applied to the partition column defeats it.

partitioning (ch4) — splitting data by a key so that work and storage divide. The decision that most affects cost and is most often made by default.

periodic snapshot (ch6) — a fact table with one row per entity per period, whether or not anything happened. Right for balances and inventory levels.

personal data (ch31) — information relating to an identifiable person. Broader than most engineers assume: an IP address, a device id, and a salted hash all qualify.

physical replication (ch7) — copying a database at the block level. Simple and exact, and it gives you a replica rather than a change stream.

PII (ch31) — personally identifiable information. A colloquial term; the regulations say "personal data" and mean something broader.

pivot (ch18) — turning rows into columns. Verbose in standard SQL, and a common interview question because the general case needs dynamic SQL.

plan (ch28) — Terraform's preview of what an apply would change. The artifact to review; applying without reading it is the whole risk.

platform engineering (ch40) — building the capabilities other teams use. The default senior path in data, and the most portable.

platform team (ch35) — the team whose product is the platform rather than the data. In a mesh transition it does not shrink; it changes to harder work.

point-in-time correctness (ch32) — using only information available at the moment being predicted. Its absence is data leakage, and the lab measures the cost at 0.830 AUC against 0.591.

Polars (ch22) — a Rust DataFrame library with a lazy API. Fast, memory-efficient, and it will lose leading zeros on an all-numeric string column exactly as pandas does.

poll loop (ch15) — a Kafka consumer's poll() cycle. Taking too long between polls gets you evicted from the group, which looks like a mysterious rebalance.

polyglot persistence (ch12) — using several storage technologies for their strengths. Correct in principle and a way to acquire five operational burdens.

polyseme (ch35) — a concept meaning subtly different things in different domains. The mesh's answer to shared entities, and it applies to a concept rather than to a fact table.

pool (ch24) — an Airflow concurrency limit shared across tasks. What stops a backfill from exhausting a database, and what a sensor can deadlock.

portfolio (ch40) — the accumulated evidence of what you have done. In this field the strongest item is a platform that reconciles.

portfolio project (ch39) — the artifact you bring to an interview. Worth more than a certification because it is checkable — and only if you can say what went wrong.

pre-authorization (ch26) — deciding in advance who may take a costly action without asking. What stops an incident waiting forty minutes for permission.

predicate pushdown (ch8) — evaluating a filter as close to the data as possible, skipping blocks whose statistics rule them out. Why Parquet statistics exist.

prefix (ch9) — the leading portion of an object key. What a listing filters on, and what partitioning manipulates.

processing time (ch4) — when a record is processed, as opposed to when the event happened. Easy to observe and almost never what a business question means.

processor (ch31) — in privacy law, a party processing data on a controller's instruction. Determines obligations; a legal determination.

producer (ch15) — a process writing to a topic. Its partitioner decides ordering and skew.

producer-side validation (ch17) — the producer checking its own output against the contract before publishing. The only place a breaking change can be stopped rather than detected.

profiles.yml (ch19) — dbt's connection configuration. Kept out of the repository, and the file that makes "it works on my machine" possible.

progressive refinement (ch34) — the medallion's core idea: each layer adds guarantees the previous one did not have.

projection (ch36) — a view of an event stream computed by folding. Pure function of the events, which is what makes replay meaningful.

projection pushdown (ch8) — reading only the columns a query needs. Free in a columnar format, and SELECT * throws it away.

promotion (ch27) — moving an artifact from one environment to the next. In ch40, the level change and the evidence it requires.

Protocol Buffers (ch11) — a compact binary format with generated code and strong schema evolution rules. Excellent for RPC and inter-service events.

provider (ch28) — a Terraform plugin for a platform. Pin its version; a minor upgrade can change a plan.

pseudonymization (ch31) — replacing direct identifiers, usually with a hash or a token. Reduces risk and changes no obligation: it is still personal data.

publication (ch14) — in Postgres logical replication, the set of tables a subscriber receives. Where a missing table becomes a silently missing stream.

purpose limitation (ch31) — data collected for one purpose may not be freely used for another. Makes a row present for some purposes, which the pipeline must know.

Q

QUALIFY (ch18) — a clause filtering on a window function's result without a subquery. Available in Snowflake, BigQuery, and DuckDB; missing in Postgres.

quarantine (ch23) — a parallel table for rows that failed an assertion, with the reason and enough context to replay. Needs a size alert and a retention, or it becomes a second landfill.

quasi-identifier (ch31) — an attribute that is not identifying alone and is identifying in combination: postcode, birth year, sex. Where re-identification actually comes from.

query plan (ch7) — the engine's chosen execution strategy. Reading one is the highest return-on-time skill in this book.

quorum (ch4) — the number of replicas that must agree. Where consistency guarantees are actually configured, usually with a default nobody chose.

R

RANK (ch18) — a window function ranking with gaps after ties. Distinct from DENSE_RANK and ROW_NUMBER, and the distinction is a common bug.

rate limit (ch16) — a cap on request frequency. Respect the server's headers rather than guessing, and back off with jitter.

raw zone (ch34) — see bronze layer.

RBAC (ch30) — role-based access control. Simpler to audit than ABAC, and it accumulates roles until nineteen of forty-seven have one member.

RDD (ch21) — Spark's original low-level API. Still available, opaque to the optimizer, and rarely the right choice now.

re-identification (ch31) — recovering an individual's identity from supposedly anonymous data. The attack k-anonymity measures resistance to.

read replica (ch7) — a copy serving reads. Relieves the primary, and it lags, which is visible to a read-after-write.

read-your-writes (ch4) — the guarantee that you see your own write immediately. Often absent on a read replica, and the source of "the update didn't save" bug reports.

rebalance (ch15) — reassigning partitions when a consumer group changes. Stops processing briefly and can replay from a committed offset.

rebuild (ch38) — recomputing the whole platform from raw. The only check that tests a property of the whole graph over time, and it costs $198.96.

reconciliation (ch13) — comparing a computed result against an independent source. The single most valuable check in this book, and only as good as the independence of its two sides.

recursive CTE (ch18) — a CTE that references itself, for hierarchies and graph walks. Powerful, and easy to write without a termination condition.

red flag (ch39) — a signal that a role or team is not what it claims. "We don't really have incidents" is the clearest.

ref (ch19) — dbt's function referencing another model. What builds the DAG, and why you never write a table name directly.

referential integrity (ch23) — every foreign key has a matching parent. Enforced by the database in OLTP and by an assertion in a warehouse.

refund netting (ch38) — subtracting refunds from the month the order was sold, not the month they settled. The rule that makes a closed month mutable.

relevance scoring (ch12) — ranking search results by estimated usefulness. BM25 and its relatives, and tuning it is a different discipline from running the index.

repartition (ch21) — redistributing data across partitions with a full shuffle. Expensive and sometimes exactly what is needed to fix skew.

replay (ch29) — reprocessing historical records from a log. The capability that makes a streaming platform recoverable, and one you do not have until you exercise it.

replay window (ch16) — how far back a source lets you re-read. Bounds your backfill regardless of what your pipeline can do.

replication (ch4) — keeping copies on several nodes. Buys durability and availability, and introduces lag and consistency questions.

replication factor (ch15) — how many copies of a Kafka partition exist. Three is the usual default, and with min.insync.replicas=2 it survives one broker.

replication lag (ch4) — how far behind a replica is. Small and non-zero, and the reason a read replica can serve stale data.

replication slot (ch7) — a Postgres structure tracking a consumer's position in the WAL. Retains WAL until consumed, so a stalled consumer fills the source's disk.

reserved instances (ch33) — committing to capacity for a discount. Real savings, and a commitment converts a future efficiency into a sunk cost.

REST API (ch16) — the most common ingestion source after a database. Pagination, rate limits, retries, and a schema that changes without notice.

restartability (ch13) — a load's ability to resume after failure without duplicating or losing. Achieved with a manifest and idempotent writes.

result cache (ch8) — a warehouse returning a previous identical query's result without computing. Cheap, and it makes naive benchmarks meaningless.

retention (ch10) — how long data is kept. Needs a period, a mechanism, and an exception path — and the mechanism is what is missing.

retroactive change (ch16) — a source altering historical records in place. Breaks incremental loads and reproducibility, and is discovered during a rebuild.

retryable error (ch16) — a failure worth attempting again: a timeout, a 429, a 503. Distinguishing it from a permanent one is most of a robust client.

revenue recognition (ch38) — the accounting rules deciding when a sale becomes revenue. Gift cards are a liability at sale, and an engineer deciding this alone is Chapter 38's third failure.

reverse ETL (ch2) — pushing warehouse data back into operational systems. Inverts your risk profile: a dashboard bug is embarrassing, a reverse-ETL bug is in front of customers.

reverse interview (ch39) — your evaluation of them. The neglected half, and questions whose answers you have not thought about are questions asked to look good.

reversible decision (ch3) — one that can be undone cheaply. Make it fast; spend the deliberation on the irreversible ones.

right to be forgotten (ch31) — the erasure right. Requires finding a person in every identifier space before deleting anything.

role explosion (ch30) — accumulating roles until they describe individuals rather than job functions. At that point they provide no abstraction and all the overhead.

role-playing dimension (ch6) — one dimension joined several times in different roles, such as order date and ship date. Implemented with views or aliases.

rollback (ch27) — reverting to a previous state. In data frequently impossible, which is why forward fixes and rehearsed rollbacks both matter.

rotation (ch26) — the on-call schedule. Its health is measured in pages per week, not in coverage.

routing (ch25) — sending an alert to the right person. The difference between an alert and a notification nobody owns.

row group (ch8) — a horizontal slice of a Parquet file, the unit statistics describe and readers skip.

row-binary format (ch11) — a row-oriented binary encoding such as Avro. Efficient for whole-record reads and wrong for column scans.

row-level security (ch31) — a policy filtering which rows a role can see, attached to the object. Auditable in one place, and it fails closed.

row-oriented storage (ch7) — storing all of a row's values together. Right for OLTP, wrong for analytics.

ROW_NUMBER (ch18) — a window function numbering rows without ties. Needs a deterministic ORDER BY, or it returns a different row each run and no uniqueness test notices.

ROWS versus RANGE (ch18) — window frame modes. They differ when the ordering column has duplicates; RANGE includes every peer and is the default.

run-length encoding (ch8) — storing a value and a count instead of repetitions. Extremely effective on sorted low-cardinality columns.

runbook (ch25) — the document telling an on-call responder what to do. Judged by whether somebody opens it at 3am, not by whether it exists.

S

salting (ch21) — adding a random component to a skewed join key so the hot key spreads across partitions. The manual fix AQE now handles automatically in many cases.

sampling (ch27) — testing against a subset of production-shaped data. Makes CI fast enough to run on every change, and it must preserve the awkward rows.

savings plan (ch33) — a spend commitment for a discount. Commit to the floor, not the current level.

scan cost (ch11) — what it costs to read data for a query. On a per-byte meter it is the cost, which is why format and layout dominate.

SCD Type 1 (ch20) — overwriting a dimension attribute on change. Simple, and it destroys history.

SCD Type 2 (ch20) — adding a new row with validity dates on change. Preserves history, and it is also a point-in-time-correct feature table.

scheduler (ch24) — the process deciding what runs when. Its own failure produces no events, which is why it needs a heartbeat rather than an alert.

schema change event (ch14) — a CDC record signalling a DDL change. Consumed rather than ignored, or the next data record surprises you.

schema drift (ch13) — a source's schema changing without notice. The most common ingestion failure and the one contracts exist to catch.

schema enforcement (ch10) — a table format rejecting writes that violate the declared schema. What makes a lake behave like a warehouse.

schema evolution (ch9) — changing a schema without rewriting data. Adding a column is free in Parquet; renaming one needs column mapping.

schema registry (ch11) — a service holding schemas and enforcing compatibility rules. Where BACKWARD and FORWARD compatibility are actually configured.

schema-on-read (ch11) — deferring schema interpretation to query time. Flexible, and it moves every type surprise to the consumer.

schema-on-write (ch11) — enforcing a schema at load time. Rejects bad data early, at the cost of rejecting data you might have wanted to keep.

scope (ch39) — the breadth of what you are responsible for. What sets level, and what to negotiate before salary.

search index (ch12) — an inverted index supporting text queries. A different data structure and a different operational burden from a database.

secondary index (ch12) — an index on a non-primary attribute. Cheap in a relational database, and a significant design decision in a distributed store.

secret (ch27) — a credential. Never in a repository, never in a log, and preferably short-lived via OIDC.

secret manager (ch28) — the service holding them. Referenced by identity rather than by another secret, or you have moved the problem.

seed (ch19) — a CSV checked into a dbt project and loaded as a table. The right home for a small, version-controlled mapping — and for the hand-made lookup table that broke a rebuild.

self-serve data platform (ch35) — capabilities that let a domain team ship without a platform engineer. More work than doing the work yourself, for a long time.

semantic layer (ch2) — where a metric is defined once and consumed everywhere. The structural answer to four definitions of active_customer.

semantic versioning (ch17) — major.minor.patch, where major signals a breaking change. Useful for a data contract precisely because it forces you to classify the change.

semi-additive measure (ch6) — summable across some dimensions and not others, such as an account balance across time. The category most often summed incorrectly.

sensitivity tier (ch30) — a classification level: public, internal, confidential. Decided by legal, propagated directionally, and enforced by the platform.

sensor (ch24) — an Airflow task that waits for a condition. Holds a worker slot unless deferrable, which is how a pool deadlocks.

sequence number (ch36) — a monotonic per-aggregate version on an event. What makes ordering checkable, and what lets a consumer discard a duplicate in one comparison.

sequential scan (ch7) — reading a whole table. Frequently faster than an index scan for a non-selective predicate, which is why the planner chooses it.

serialization (ch11) — converting data to bytes for storage or transport. The format decision that determines compression, splittability, and schema evolution.

serve (ch2) — the lifecycle stage where data reaches a consumer. The entire user interface of your work, and systematically under-invested in.

service level agreement (ch17) — a promise about a dataset's timeliness or quality, with consequences. Distinct from an SLO, which is internal.

session window (ch29) — a window defined by a gap in activity rather than by a clock. Requires state per key and is neither commutative nor last-write-wins.

sessionization (ch18) — grouping events into sessions by an inactivity gap. A gaps-and-islands problem, and Kestrel's is the single largest job in the platform.

set-based thinking (ch18) — expressing a transformation as an operation on sets rather than a loop. The mental shift that makes SQL productive.

SettingWithCopyWarning (ch22) — pandas warning that an assignment may not affect the original object. Almost always a real bug, and almost always ignored.

severity (ch23) — how badly a failed assertion matters: blocking, warning, or note. The classification that keeps a blocking set small enough that nobody disables it.

shadow deploy (ch27) — running new code alongside old without acting on its output. The data equivalent of a canary, and the only one that compares values.

shadow running (ch37) — running a legacy and a replacement system in parallel and reconciling. Where 687 differences got classified, 196 of them the legacy system's fault.

shadow schema (ch17) — building a contract's proposed change into a parallel schema to test consumers against. Expand-contract's mechanism.

sharding (ch12) — splitting data across independent database instances. Scales writes, and every cross-shard query becomes your problem.

showback (ch33) — reporting each team's spend without moving budget. The step before chargeback, and usually the one that does the work.

shuffle (ch21) — redistributing data across the network so that related records meet. The expensive operation, and the one every join and aggregation may require.

signal (ch39) — evidence an interviewer is looking for. "Have you operated something" is the strongest, and it shows up in what you mention breaking.

silver layer (ch34) — typed, deduplicated, conformed — and encoding no decision anyone could disagree with. The test: if two competent people could disagree, it is a business rule.

singular test (ch19) — a dbt test written as a SQL query returning failing rows. The escape hatch for anything a generic test cannot express.

skew (ch4) — uneven distribution of work across partitions. High cardinality does not prevent it, and one key at 8.1% turned a 22-minute job into 71.

SLA (ch1) — the promise. Kestrel's is daily_revenue by 06:00 America/New_York.

SLA miss (ch24) — a run finishing later than its agreed deadline. Worth alerting on slack remaining rather than on the miss.

SLI (ch26) — a service level indicator: the measured quantity an SLO is about. Choose one a consumer would recognize.

sliding window (ch29) — overlapping windows advancing by less than their length. More output than tumbling, and each event lands in several windows.

slim CI (ch19) — building only modified models and their descendants, using deferral and state comparison. The difference between a CI that runs and one that is disabled.

SLO (ch25) — a service level objective: a target for an SLI, with an error budget. Internal, and the number an on-call rotation is designed around.

slot (ch8) — BigQuery's unit of compute. In Postgres (ch7), a replication slot — an unrelated and easily confused term.

slowly changing dimension (ch20) — a dimension whose attributes change over time. Types 1 through 6, of which 1 and 2 cover almost everything.

small file problem (ch33) — many tiny files inflating request costs and task overhead. Real, and its cost justification is usually much weaker than its latency one.

small files problem (ch9) — the same thing, at the storage layer.

snappy (ch11) — a fast, splittable compression codec. Parquet's usual default, and the right choice unless you have measured otherwise.

snapshot (ch7) — a consistent point-in-time view. In dbt (ch19), the mechanism that builds SCD Type 2 history from a mutable source.

snowflake schema (ch6) — a star schema with normalized dimensions. Saves storage that costs nothing and adds joins that cost something.

soft delete (ch13) — flagging a row as deleted rather than removing it. Visible to an incremental load, which is why it is kinder to downstream systems.

sort key (ch12) — the attribute rows are ordered by within a partition. Determines which range queries are cheap.

sort order (ch11) — the physical ordering of data in a file or table. Determines compression ratio and which predicates can skip.

sort-merge join (ch21) — sorting both sides and merging. Spark's fallback when a broadcast is not possible, and the reason a shuffle appears in the plan.

source (ch19) — in dbt, a declared raw table with freshness expectations. Where the lineage graph starts.

source fidelity (ch34) — bronze's guarantee: exactly what the source sent, unmodified. What makes "it arrived that way" a provable statement rather than a claim.

source freshness (ch19) — dbt's check that a source table is no older than a threshold. The assertion that catches an upstream problem before your models run on stale data.

source of truth (ch1) — the system whose value is authoritative. Naming it explicitly is what stops two systems both being right.

source system (ch2) — where data originates, usually outside your control. The stage with the most leverage and the least authority.

specialization (ch40) — a track: streaming, platform, analytics engineering, ML infrastructure, governance. Each trades something, and the two columns nobody reads are on-call and obsolescence.

spill (ch8) — writing intermediate data to disk when memory is exhausted. Slow, and a sign that a partition is too large or a join too wide.

splittable (ch11) — a file a reader can start consuming from the middle of. gzip is not; snappy in Parquet is, which is why a single large gzip file becomes a single task.

spot instances (ch33) — heavily discounted capacity that can be reclaimed with short notice. Excellent for a re-runnable batch job, unusable for a deadline that cannot be missed twice.

SQL round (ch39) — the interview testing whether you notice the awkward rows, not whether you know window functions.

staff engineer (ch40) — the level whose scope is the platform, whose output is writing and conversations, and whose best work is often invisible.

stage (ch21) — a set of Spark tasks executable without a shuffle. Stage boundaries are shuffle boundaries.

staging (ch27) — a pre-production environment. In data the hard part is that it cannot hold production data, which is Chapter 31's largest exposure.

staging model (ch19) — a dbt model doing one-to-one cleanup of a source: renaming, casting, deduplicating. Silver, in dbt's vocabulary.

STAR (ch39) — situation, task, action, result. A behavioral-answer structure that survives follow-ups if you add a number.

star schema (ch6) — a fact table surrounded by denormalized dimensions. The default warehouse shape, and correct far more often than its critics allow.

state comparison (ch27) — comparing an artifact against a previous run to find what changed. What makes slim CI possible.

state file (ch28) — Terraform's record of what it manages. Remote, locked, and separated by blast radius.

state store (ch29) — a stream processor's durable per-key state. Grows without a TTL, and its size is the thing that fails at 3am.

statement timeout (ch7) — a limit on query duration. The cheapest protection against one query consuming a database.

StatsD (ch25) — a simple metrics protocol. Old, widely supported, and adequate for pipeline counters.

steward (ch30) — a person accountable for a dataset's meaning, as distinct from its pipeline. Useful when they can act; a name on a page when they cannot.

storage class (ch9) — S3's tiers by access frequency. Cheaper storage and more expensive retrieval, and a recommendation that omits retrieval cost is worse than none.

storage-compute separation (ch8) — scaling storage and compute independently. The property that makes cloud warehouses elastic and makes idle compute a pure waste.

store (ch2) — the lifecycle stage where data rests. Where format, layout, and retention decisions are made, mostly by default.

stored procedure (ch37) — logic held in the database. Readable and entangled with transaction management, error handling, and business rules nobody knew were there.

strangler fig (ch37) — growing a new system around an old one until the old one does nothing. The default migration pattern, and step six — actually deleting it — is the one that does not happen.

stream processing (ch3) — computing continuously over unbounded data. Necessary less often than the literature implies, and the four questions in Chapter 29 decide.

streaming (ch29) — see stream processing. Also the specialization with the deepest technical content and the worst on-call, which are the same fact.

streaming execution (ch22) — processing data in chunks that fit memory. What lets DuckDB and Polars handle files larger than RAM.

structured logging (ch25) — emitting logs as key-value records rather than prose. Queryable, and the right place to refuse to serialize a field tagged as personal data.

subject access request (ch31) — see DSAR.

surrogate key (ch6) — a meaningless integer key for a dimension row. Stable across source changes, and it must not leak into a reconciliation.

synthetic data (ch27) — generated data standing in for production. Safe, and only as good as the awkward cases you thought to generate.

system design interview (ch39) — the main event. Requirements are 25% of the score and the diagram is 10%, and most candidates invert that.

T

table format (ch10) — Iceberg, Delta, or Hudi: metadata over files giving atomicity, schema evolution, time travel, and row-level deletes.

tagging (ch28) — labelling resources with team and purpose. The mechanism cost attribution depends on, and it fixes the future and nothing else.

take-home (ch39) — an interview exercise done on your own time. The round with the widest quality range, and the most expensive one for you.

target (ch19) — a dbt connection profile: which warehouse, which schema, which credentials.

target leakage (ch32) — a feature containing information about the label that would not exist at prediction time. Data leakage's most common form.

task (ch21) — the smallest unit of Spark work, one partition of one stage. In Airflow (ch24), one node of a DAG.

task group (ch24) — an Airflow construct for organizing tasks visually and logically. Cosmetic and genuinely helpful at scale.

TaskFlow API (ch24) — Airflow's decorator-based authoring style. Less boilerplate, and ast.walk will descend into nested @task functions if you write a linter.

technical currency (ch40) — being up to date. Achieved by learning one tool per category deeply, not five shallowly.

technical debt (ch37) — the accumulated cost of past shortcuts. In data it is usually a rule nobody wrote down rather than code nobody refactored.

technical metadata (ch30) — types, sizes, partitions, lineage. Cheap to crawl, and it is not what makes a catalog useful.

technical screen (ch39) — the first technical round, usually SQL. The widest cut after the résumé.

telemetry (ch25) — the metrics, logs, and traces a system emits. And a place personal data escapes a residency boundary.

terminal error (ch16) — a failure that will not succeed on retry: a 400, a 404, an auth failure. Retrying it wastes quota and hides the problem.

Terraform (ch28) — the dominant infrastructure-as-code tool. Declarative, with a plan you can review, and a state file you must protect.

testing the data (ch23) — asserting properties of the output. What catches a pipeline that succeeded and produced wrong numbers.

testing the pipeline (ch23) — asserting that the code behaves. Necessary, insufficient, and what most teams have.

third normal form (ch6) — a normalization level removing transitive dependencies. Right for an operational schema; a source of joins in a warehouse.

threads (ch19) — dbt's parallelism setting. Bounded by the warehouse's concurrency, not by your machine.

threshold (ch23) — the value at which an assertion fails. Absolute thresholds miss relative regressions, which is how a 3.75x slowdown passes an 8-hour alarm.

throughput (ch3) — volume processed per unit time. Trades against latency, and both are usually quoted without the other.

thundering herd (ch16) — many clients retrying simultaneously and overwhelming a recovering service. Prevented by jitter, not by backoff alone.

time travel (ch10) — querying a table as of a past version. A table format feature, a debugging superpower, and a privacy problem (ch31).

time-series database (ch12) — a store optimized for timestamped metrics. Excellent within its model, and adding a high-cardinality label produces billions of series.

timeline (ch26) — the ordered record of an incident. The most valuable postmortem artifact and the one hardest to reconstruct afterwards, so write it during.

timeliness (ch23) — whether data arrived when expected. Freshness, expressed as a quality dimension.

timestamp strategy (ch20) — an incremental check based on updated_at. The most common, and it misses hard deletes and retroactive changes.

timestamp with time zone (ch7) — Postgres' timestamptz. Stores an instant; the session timezone decides how a boundary comparison behaves, which is Chapter 38's first failed reconciliation.

toil (ch26) — manual, repetitive operational work that scales with the system. The thing an SRE practice exists to bound, and above about half it is why people leave.

token bucket (ch16) — a rate-limiting algorithm allowing bursts up to a capacity. What most APIs implement, so matching it client-side is more effective than a fixed delay.

token refresh (ch16) — exchanging a refresh token for a new access token. A step that must be thread-safe and must not be logged.

tokenization (ch31) — replacing a value with a random token, with the mapping in a vault. Everything a hash gives you plus reversibility, and a vault on the critical path.

tombstone (ch10) — a marker recording a deletion. In Kafka, a null-valued record that removes a key under compaction.

topic (ch15) — a named Kafka stream, divided into partitions.

total cost of ownership (ch5) — licence plus infrastructure plus the engineering time to operate it. The third term dominates and is the one omitted from comparisons.

trace (ch25) — a record of a request's path through a system. Rarer in data than in services, and useful across a multi-hop ingestion path.

trailing median (ch25) — a baseline computed from recent history rather than a fixed value. What makes a volume band adapt to growth without adapting to a defect.

training/serving skew (ch32) — the same feature computed differently in two code paths. 5.37% overall at Kestrel, and 98% among the customers the model was most confident about.

transaction boundary (ch14) — the grouping of CDC events belonging to one source transaction. Preserving it is what stops a consumer seeing half an update.

transaction isolation (ch7) — how concurrent transactions see one another. The setting whose default you have not read and whose anomalies you have blamed on something else.

transaction log (ch10) — the ordered record of changes to a table. Delta's _delta_log, Postgres' WAL, and the thing CDC reads.

transactional outbox (ch36) — writing the row and the event in one database transaction, with a relay publishing from the outbox. Trades loss for duplication, which is the right trade.

transactional sink (ch29) — a destination supporting atomic commits, so a stream can write exactly-once within its boundary.

transform (ch2) — the lifecycle stage where data is shaped for use. Where the business's rules live and where they get decided by accident.

transformation framework (ch5) — dbt or an equivalent. Brings version control, tests, and lineage to SQL, which is most of what a platform needs.

transitive compatibility (ch17) — compatibility across all previous versions, not just the last one. The strictest schema-registry mode.

trigger (ch24) — what causes a DAG run: a schedule, a dataset update, or a manual invocation.

TTL (ch12) — time to live: automatic expiry. The mechanism that makes a state store or a cache bounded.

tumbling window (ch29) — fixed, non-overlapping windows. The simplest windowing and usually the right one.

U

undercurrent (ch2) — a concern spanning every lifecycle stage: security, data management, DataOps, architecture, orchestration, and software engineering.

undocumented dependency (ch37) — an input a job needs that is not named in any configuration. Found by running the job, and 36 of them across Kestrel's twelve legacy jobs.

uniqueness (ch23) — no duplicates on a key. The assertion that catches fan-out, and the one that passes while a non-deterministic dedup returns a different row each run.

unit economics (ch33) — cost per business unit. $0.1526 per order, 0.201% of GMV — and a total answers no question anyone has.

unit test (ch19) — a test of one transformation with fixed inputs and expected outputs. dbt's version arrived late and is the right tool for a gnarly CASE expression.

unknown member (ch19) — a placeholder dimension row for facts whose dimension has not arrived. Keeps the join inner and the counts right.

unpivot (ch18) — turning columns into rows. Usually easier than pivoting and equally verbose.

unplanned work (ch40) — engineering time consumed by interrupts. Above about half it is why people leave, and it is the resting state without deliberate effort.

upcasting (ch36) — converting old event versions to the current shape at read time, in one place. Where schema evolution actually lives in an event-sourced system.

updated_at (ch13) — the column most incremental loads depend on, and which lies in four distinct ways: not set on every path, set by the application clock, unchanged by bulk updates, and absent on deletes.

upsert (ch20) — insert or update. See merge.

V

vacuum (ch7) — Postgres' process reclaiming dead tuples. Autovacuum handles most cases, and a long-running transaction defeats it.

valid_from (ch20) — the start of an SCD Type 2 row's validity. Half of what makes an as-of join possible.

valid_to (ch20) — the end. Null or a sentinel for the current row, and the choice matters for range predicates.

validity (ch23) — whether a value conforms to its expected format or domain. The easiest quality dimension to assert and the least likely to be the actual problem.

vector database (ch12) — a store for embeddings with approximate nearest-neighbour search. A real capability, and frequently a feature of a database you already run.

vectorization (ch22) — operating on batches of values rather than one at a time. The reason a columnar engine is an order of magnitude faster than a row-at-a-time loop.

vendor lock-in (ch5) — the cost of leaving. Real, usually overstated for storage and understated for proprietary transformation logic.

verification (ch37) — confirming a migration produces the same result. A parallel run and a reconciliation, and its absence is why "we migrated it" is a claim.

virtual warehouse (ch8) — Snowflake's compute cluster. Sized independently of storage, billed by the second while running, and the thing to auto-suspend.

volume (ch25) — how much data arrived. Monitored against a band, and a band set from ordinary days fires every Black Friday.

volume floor (ch23) — a minimum row count below which a load is treated as failed. Catches a truncated source, which a percentage band can miss.

W

watermark (ch2) — an assertion that no more events older than a timestamp will arrive. The mechanism that lets a stream close a window, and always wrong sometimes.

wide transformation (ch21) — a Spark operation requiring a shuffle. The expensive kind.

wide-column store (ch12) — Cassandra and its relatives: rows with dynamic columns, partitioned by key. Excellent write throughput, and the query patterns must be known in advance.

window function (ch18) — a function computing over a set of rows related to the current one, without collapsing them. The single most valuable SQL feature for data engineering.

windowing (ch29) — grouping unbounded stream data into finite chunks. Tumbling, sliding, or session, and the choice follows the question.

worker (ch24) — the process executing an Airflow task. Its slots are the resource a sensor can hold.

workload identity (ch28) — a cloud identity attached to a running workload rather than to a stored key. Removes the long-lived credential entirely.

workspace (ch28) — a Terraform environment sharing configuration with separate state.

write-ahead log (ch14) — the durability mechanism of a database, and the source CDC reads. Postgres' WAL, MySQL's binlog.

X

XCom (ch24) — Airflow's mechanism for passing small values between tasks. Small: it is stored in the metadata database, and it is not a data transport.

Z

Z-ordering (ch10) — a multi-dimensional clustering technique co-locating rows similar across several columns. Improves skipping when queries filter on more than one dimension.

zero-copy (ch22) — passing data between systems without serializing or copying. Arrow's central promise, and what makes engine interop cheap.

zone map (ch8) — per-block min/max statistics used to skip data. Parquet's row-group statistics are the same idea under a different name.

zstd (ch11) — a modern compression codec with better ratios than snappy at comparable speed. Usually the right choice today, and check that every reader in your stack supports it.